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DHS has largely defined but has not adequately implemented the full range of 
controls that is reflected in relevant guidance and related best practices and is 
needed to effectively manage and oversee its SBlnet prime contractor. To the 
department's credit, it has defined a number of key policies and procedures 
for verifying and accepting contract deliverables and conducting technical 
reviews, such as the criteria that need to be met before commencing and 
concluding a critical design review. Moreover, it has implemented some of 
these defined practices, such as those associated with training key contractor 
management and oversight officials. However, DHS has not effectively 
implemented other controls. For example, it has not adequately documented 
its review of contract deliverables and communicated to the prime contractor 
the basis for rejecting certain deliverables. Further, it has not ensured that key 
documentation satisfied relevant criteria before concluding key technical 
reviews. These weaknesses can be attributed in part to limitations in the 
defined verification and acceptance deliverable process, a program office 
decision to exclude certain deliverables from the process, and insufficient 
time to review technical review documentation. All told, DHS has not 
effectively managed and overseen its SBlnet prime contractor, thus resulting 
in costly rework and contributing to SBlnet's well-chronicled history of not 
delivering promised capabilities and benefits on time and within budget. 

DHS has not effectively monitored the SBlnet prime contractor's progress in 
meeting cost and schedule expectations. While DHS has used earned value 
management (EVM), which is a proven management approach for 
understanding program status and identifying early warning signs of 
impending schedule delays and cost overruns, it has not ensured that its 
contractor has effectively implemented EVM. In particular, DHS has not 
ensured that validated performance baselines (the estimated value of planned 
work against which performance is measured) were timely, complete, and 
accurate. For example, the contractor was allowed to perform work on task 
orders for several months without a validated baseline in place. Further, not 
all work to be performed was included in the baselines that were eventually 
established, and the schedules for completing this work did not substantially 
comply with six of the eight key practices that relevant guidance states are 
important to having a reliable schedule. Also, DHS regularly received 
incomplete and anomalous EVM data from the prime contractor, which it had 
to rely on to measure progress and project the time and cost to complete the 
program. As a result, DHS has not been able to gain meaningful and proactive 
insight into potential cost and schedule performance shortfalls, and thus take 
corrective actions to avoid shortfalls in the future. Program officials attributed 
these weaknesses in part to the instability in the scope of the work to be 
performed, and the contractor's use of estimated, rather than actual, costs for 
subcontractor work and the subsequent adjustments that are made when 
actual costs are received. This inability has contributed to the program's 
failure to live up to expectations and to it costing more and taking longer than 
was necessary. 
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United States Government Accountability Office 
Washington, DC 20548 



October 18, 2010 

The Honorable Bennie G. Thompson 
Chairman 

Committee on Homeland Security 
House of Representatives 

The Honorable Christopher P. Carney 
Chairman 

The Honorable Gus M. Bilirakis 
Ranking Member 

Subcommittee on Management, Investigations, 

and Oversight 
Committee on Homeland Security 
House of Representatives 

The Honorable Mike Rogers 
Ranking Member 

Subcommittee on Emergency Communications, 

Preparedness, and Response 
Committee on Homeland Security 
House of Representatives 

To enhance border security and reduce illegal immigration, the 
Department of Homeland Security (DHS) launched its multiyear, 
multibillion dollar Secure Border Initiative (SBI) program in November 
2005. Through SBI, DHS intends to enhance surveillance technologies, 
increase staffing levels, enforce immigration laws, and improve the 
physical infrastructure along our borders. Within SBI, the Secure Border 
Initiative Network (SBlnet) is a multibillion-dollar program that involves 
the acquisition, development, integration, deployment, and operation and 
maintenance of surveillance technologies to create a "virtual fence" along 
the border as well as command, control, communications, and intelligence 
(C3I) technologies to create a picture of the border in command centers 
and vehicles. 

DHS's strategy is to deliver SBlnet capabilities incrementally. In doing so, 
the department has relied heavily on its prime contractor — the Boeing 
Company. Because of the importance of effective contractor management 
and oversight to the successful deployment and operation of SBlnet 
capabilities, you asked us to determine to what extent DHS (1) has defined 
and implemented effective controls for managing and overseeing the 
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SBlnet prime contractor and (2) is effectively monitoring the prime 
contractor's progress in meeting cost and schedule expectations. 

To accomplish our objectives, we focused on DHS's management and 
oversight of four key task orders associated with designing, developing, 
and deploying the first increment of SBlnet, known as Block 1, to its initial 
deployment sites in Arizona — the Tucson Border Patrol Station (TUS-1) 
and the Ajo Border Patrol Station (AJO-1). In doing so, we reviewed DHS 
and program guidance, policies, and plans for managing and overseeing 
the contractor; task orders and associated modifications; contract 
deliverable review comments; contractor schedules; the work breakdown 
structure governing all task orders; integrated baseline review documents 
for each task order; and task order contract cost performance reports. We 
also analyzed a nonprobability, random sample of 28 contract deliverables; 
a nonprobability, random sample of 8 technical reviews conducted with 
the contractor; and a nonprobability sample of 11 action items identified 
during program management reviews. 1 In addition, we interviewed 
program officials about contractor management and oversight activities, 
such as verifying and accepting contract deliverables and conducting 
technical and management reviews. We also interviewed program officials 
about the prime contractor's cost and schedule performance. We then 
compared this information to relevant guidance and leading industry 
practices on contractor management and oversight, schedule estimating, 
and earned value management (EVM), 2 such as the Software Engineering 
Institute's (SEI) Capability Maturity Model Integration® (CMMI®) for 
Acquisition, GAO's Cost Estimating and Assessment Guide, and the 
American National Standards Institute (ANSI) standard. 3 



x We also selected one additional contract deliverable and one additional technical review 
for review. 

2 EVM is a project management approach that, if implemented appropriately, provides 
objective reports of project status, produces early warning signs of impending schedule 
delays and cost overruns, and provides unbiased estimates of a program's total costs. 

3 SEI, CMMf for Acquisition, Version 1.2 (Pittsburgh, Pa., November 2007); and GAO, GAO 
Cost Estimating and Assessment Guide: Best Practices for Developing and Managing 
Capital Program Costs, GAO-09-3SP (Washington, D.C.: March 2009). Also, recognizing the 
importance of ensuring quality earned value data, ANSI and the Electronic Industries 
Alliance (EIA) jointly established a national standard for EVM systems in May 1998 
(ANSI/EIA-748-A-1998). This standard, commonly called the ANSI standard, is comprised of 
guidelines on how to establish a sound EVM system. This document was updated in July 
2007 and is referred to as ANSI/EIA-748-B. 
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We conducted this performance audit from June 2009 to October 2010 in 
accordance with generally accepted government auditing standards. Those 
standards require that we plan and perform the audit to obtain sufficient, 
appropriate evidence to provide a reasonable basis for our findings and 
conclusions based on our audit objectives. We believe that the evidence 
obtained provides a reasonable basis for our findings and conclusions 
based on our audit objectives. Further details of our objectives, scope, and 
methodology are in appendix I. 



Background 



Managed by U.S. Customs and Border Protection (CBP), SBlnet is to 
strengthen DHS's ability to detect, identify, classify, track, and respond to 
illegal breaches at and between ports of entry. It includes the acquisition, 
development, integration, deployment, and operations and maintenance of 
a mix of surveillance technologies, such as cameras, radars, sensors, and 
C3I technologies. Surveillance technologies include unattended ground 
sensors that can detect heat and vibrations associated with foot traffic and 
metal associated with vehicles, radar mounted on fixed towers that can 
detect movement, and cameras mounted on fixed towers that can be used 
to identify and classify items of interest detected and tracked by ground 
sensors and radar. These technologies are generally commercial off-the- 
shelf products. C3I technologies include customized software 
development to produce a common operating picture (COP) — a uniform 
presentation of activities within specific areas along the border — as well 
as the use of CBP network capabilities. Together, the surveillance 
technologies are to gather information along the border and transmit this 
information to COP terminals located in command centers, which are to 
assemble it to provide CBP agents with border situational awareness, to, 
among other things, enhance agents' tactical decisionmaking regarding 
potential apprehensions. 

Since fiscal year 2006, DHS has received about $4.4 billion in 
appropriations for SBI, including about $2.5 billion for physical fencing 
and related infrastructure, about $1.5 billion for virtual fencing (e.g., 
surveillance systems) and related infrastructure (e.g., towers), and about 
$300 million for program management. 4 As of May 2010, DHS had 
obligated about $1.02 billion for SBlnet. 



4 The remaining $126 million was for upgrading CBP telecommunications links. 
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Program and Acquisition 
Organizational Structure 



The SBI Program Management Office, which is organizationally within 
CBP, is responsible for managing key acquisition functions associated 
with SBlnet, including prime contractor management and oversight. It is 
organized into four components: the SBlnet System Program Office (SPO), 
Operational Integration Division, Business Operations Division, and 
Systems Engineering Division. 5 The SPO is responsible for such key 
contractor oversight activities as verifying and accepting contract 
deliverables and conducting contractor management and technical 
reviews. In addition, the Business Operations Division has primary 
responsibility for monitoring the prime contractor's cost and schedule 
performance. As of May 2010, the SBI Program Management Office was 
staffed with 194 people — 105 government employees, 78 contractor staff, 
and 11 detailees. (See figure 1 for a partial SBI program organization 
chart.) 



Figure 1 : Partial High-Level Depiction of SBI Program Organization 
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Source: GAO analysis based on DHS data. 



In addition, CBP has engaged the Defense Contract Management Agency 
to, among other things, perform surveillance of the prime contractor's 



The physical infrastructure (e.g., physical fencing) portion of SBI is managed on a day-to- 
day basis by CBP's Office of Finance Facilities Management and Engineering division. 
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EVM, systems engineering, hardware and software quality assurance, and 
risk management. 



Further, the SBI Contracting Division, which is organizationally within the 
CBP Office of Administration Procurement Directorate's Enterprise 
Contracting Office, is responsible for performing contract administration 
activities, such as maintaining the contract file and notifying the 
contractor in writing of whether deliverables are accepted or rejected. The 
SBI Contracting Division is organized into three branches: SBI 
Contracting, SBlnet System Contracting, and SBlnet Deployment 
Contracting. As of May 2010, the SBI Contracting Division was staffed with 
22 people — 18 government employees and 4 contractor staff. (See figure 2 
for a partial SBI Contracting Division organization chart.) 

Figure 2: Partial High-Level Depiction of SBI Contracting Division Organization 
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Overview of SBInet 
Acquisition Strategy and 
Program Status 



In September 2006, CBP awarded an Indefinite Delivery/Indefinite 
Quantity (IDIQ) 6 prime contract to the Boeing Company. The contract was 
for 3 base years with three additional 1-year options for designing, 
producing, testing, deploying, and sustaining SBI. In September 2009, CBP 
exercised the first option year. Under the prime contract, CBP has issued 
11 task orders that relate to SBInet, covering for example, COP design and 
development, system deployment, and system maintenance and logistics 
support. As of May 2010, 5 of the 11 task orders were complete and 6 were 
ongoing. (See table 1 for a summary of the SBInet task orders.) 



Table 1: Summary of SBInef Task Orders as of May 2010 



(Dollars in millions) 


Task order Description 


Period of 
performance 


Approximate 
contract 
value 


Approximate 
contract 
obligation 


Contract type 


Completed task orders 


Program Mission engineering, 
Management facilities and 
infrastructure, 


Sept. 2006-Apr. 
2008 


$146.9 


$146.9 


Cost-plus-fixed- 
fee 3 



systems engineering, 
test and evaluation, 
and program 
management 
services. 

Project 28 Prototype along 28 Oct. 2006-Feb. 2008 20.7 20.7 Firm-fixed-price 

miles of the border in 
the Tucson sector. 



Project 28 Project 28 operational Dec. 2007-Dec 

Contractor maintenance and 2009 

Maintenance logistics support. 

Logistics and 
Support 

Design for Buffalo SBInef design of Feb. 2009-July 2009 0.6 0.6 Firm-fixed-price" 

Sector remote video 

surveillance system 

capability for the 

Buffalo sector. 



10.6 



10.6 



Cost-plus-fixed- 
fee 



IDIQ contracts are used when the government cannot predetermine, above a specified 
minimum, the precise quantities of supplies or services that it will require during the 
contract period. Under IDIQ contracts, the government can issue delivery orders (for 
supplies) and task orders (for services). 
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Approximate 


Approximate 






Period of 


contract 


contract 


Task order 


Description 


performance 


value 


obligation Contract type 



System 



Program 

management and 
systems engineering 
activities required to 
integrate all task 
orders. 



Apr. 2008-May 2010 



222.1 



215.9 Cost-plus-award- 
fee 



Ongoing task orders 



Command, 
Control, 

Communications, 
and Intelligence 
(C3I) Common 
Operating Picture 
(COP) 



Design, development, 
demonstration, and 
operations and 
maintenance of a 
functional C3I/COP 
system. 



Dec. 2007-June 
2010 



70S 



79.3 Cost-plus-award- 
fee/cost-plus- 
fixed-fee/firm- 
fixed-price d 



Design 



Design of deployment 
solution, 
environmental 
clearance support, 
and other location- 
related work for the 
Tucson sector. 



Aug. 2007-July 2010 



115.0 



115.0 Cost-plus-fixed- 
fee 



Arizona 
Deployment 



Deployment to two 
sites covering 
approximately 53 
miles of the southwest 
border in the Tucson 
sector 



June 2008-May 
2011 



148.5 



148.5 Cost-plus- 

incentive-fee/cost- 
plus-award-fee e 



Integrated 
Logistics Support 



Maintenance logistics 
and operational 
support. 



Aug. 2008-Sept. 
2010 



61.6 



61.6 Cost-plus-fixed- 
fee* 



Northern Border 
Project 



Design, installation, 
and deployment of 
surveillance 
capabilities in the 
Detroit and Buffalo 
sectors. 



Mar. 2009-Dec. 
2010 



22.7 



22.7 Fixed-price 



Systems Support 



Program 

management and 
systems engineering 
activities required to 
integrate all task 
orders; C3I/COP 
maintenance. 



May 2010-Mar. 2011 



28.6 



28.6 Cost-plus-award- 
fee 



Total 



$847.80 



$850.40 



Source: GAO analysis of DHS data. 
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Note: "Fixed-price" types of contracts provide for a firm price or, in appropriate cases, an adjustable 
price; "firm-fixed-price" contracts provide for a price not subject to adjustment on the basis of the 
contractor's experience in performing the contract; "cost-plus-incentive-fee" contracts provide for the 
reimbursement of allowable costs plus an initially negotiated fee, to be adjusted later based on the 
relationship of total allowable costs to total target costs; "cost-plus-award-fee" contracts provide for 
the reimbursement of allowable costs plus a base fee fixed at the contract's inception (which may be 
zero) and an award amount that the government determines to be sufficient to motivate excellence in 
performance; and "cost-plus-fixed-fee" contracts provide for the reimbursement of allowable costs 
plus a negotiated fee fixed at the inception of the contract. 

"The initial contract type of the task order was a cost-plus-award-fee. A final award fee determination 
did not take place because of scope and schedule changes. In lieu of the final award fee 
determination, the contract type was changed to a cost-plus-fixed-fee. 

"The travel component of the task order is cost reimbursable. 

According to SBI Contracting Division officials, the contract value with the cost overruns is $79.3 
million. 

"The initial contract type of the task order was cost-plus-award-fee. On July 31 , 2009, additional 
development work was definitized as a cost-plus-fixed-fee structure. Further, on September 30, 2009, 
the software operations and maintenance component of the task order was changed to a firm-fixed- 
price structure. 

"The initial contract type of the task order was a cost-plus-incentive-fee. On November 20, 2009, the 
performance and schedule incentives component of the task order was changed to a cost-plus- 
award-fee. The cost incentives component remains a cost-plus-incentive-fee structure. 

'The initial contract type of the task order was cost-plus-incentive-fee. On November 6, 2009, future 
work under the task order was changed to a cost-plus-fixed-fee structure. 

Through the task orders, CBP's strategy is to deliver SBlnet capabilities 
incrementally. To accomplish this, the program office adopted an 
evolutionary system life cycle management approach in which system 
capabilities are to be delivered to designated locations in a series of 
discrete subsets of system functional and performance capabilities that 
are referred to as blocks. The first block (Block 1) includes the purchase 
of commercially available surveillance systems, development of 
customized COP systems and software, and use of existing CBP 
communications and network capabilities. According to program officials, 
as of July 2010, the Block 1 design was deployed to TUS-1 and was being 
deployed to AJO-1, both of which are located in CBP's Tucson Sector of 
the southwest border. Together, these two deployments cover 53 miles of 
the 1,989-mile-long southern border. Also, according to program officials, 
as of July 2010, TUS-1 and AJO-1 were to be accepted in September 2010 
and December 2010, respectively. 

In January 2010, the Secretary of Homeland Security ordered an 
assessment of the SBI program. In addition, on March 16, 2010, the 
Secretary froze fiscal year 2010 funding for any work on SBlnet beyond 
TUS-1 and AJO-1 until the assessment is completed. Also at that time, the 
Secretary redirected $35 million that had been allocated to SBlnet Block 1 
deployment to other tested and commercially-available technologies, such 
as truck-mounted cameras and radar, called Mobile Surveillance Systems, 
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to meet near-term needs. 7 According to the SBI Executive Director, the 
department's assessment would consist of a comprehensive and science- 
based assessment to determine if there are alternatives to SBlnet that may 
more efficiently, effectively, and economically meet U.S. border security 
needs. Further, the Executive Director stated that if the assessment 
suggests that the SBlnet capabilities are worth the cost, DHS will extend 
its deployment to sites beyond TUS-1 and AJO-1. However, if the 
assessment suggests that alternative technology options represent the best 
balance of capability and cost-effectiveness, DHS will redirect resources to 
these other options. According to program officials, the initial phase of the 
assessment, which addresses the Arizona border, was completed in July 
2010, and the results are currently being reviewed by senior DHS 
management. Officials were unable to provide a date for completion of the 
review. 



Use of Acquisition 
Management Disciplines Is 
Important to Programs 
Like SBlnet 



Our research and evaluations of information technology programs have 
shown that delivering promised system capabilities and benefits on time 
and within budget largely depends on the extent to which key acquisition 
management disciplines are employed. These disciplines include, among 
others, requirements management, risk management, and test 
management. As is discussed in the following section, we have previously 
reported on the extent to which these disciplines have been employed on 
SBlnet. 



Contractor management and oversight, which is the focus of this report, is 
another acquisition management discipline. Among other things, this 
discipline helps to ensure that the contractor performs the requirements of 
the contract and the government receives the service or product intended. 
DHS acquisition policies and guidance, along with other relevant guidance, 
recognize the importance of these management and oversight disciplines. 
As we have previously reported, not employing them increases the risk 



7 The $35 million was part of American Recovery and Reinvestment Act of 2009 funding. 
Pub. L. No. 111-5, 123 Stat. 115, 162 (Feb. 17, 2009). 
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that system acquisitions will not perform as intended and will require 
expensive and time-consuming rework. 8 



Since 2007, we have identified a range of management weaknesses and 
risks facing SBlnet, and we have made a number of recommendations to 
address them that DHS has largely agreed with, and to varying degrees, 
taken actions to address. For example, in February 2007, we reported that 
DHS had not fully defined activities, milestones, and costs for 
implementing the program or demonstrated how program activities would 
further the strategic goals and objectives of SBI. 9 Further, we reported that 
the program's schedule contained a high level of concurrency among 
related tasks and activities, which introduced considerable risk. 
Accordingly, we recommended that DHS define explicit and measurable 
commitments relative to, among other things, program capabilities, 
schedules, and costs, and re-examine the level of concurrency in the 
schedule and adjust the acquisition strategy appropriately. 

In September 2008, we reported that a number of program management 
weaknesses put SBlnet at risk of not performing as intended and taking 
longer and costing more to deliver than was necessary. 10 Specifically, we 
reported the following: 

• Important aspects of SBlnet were ambiguous and in a continued state of 
flux, making it unclear and uncertain what technology capabilities were to 
be delivered when. For example, the scope and timing of planned SBlnet 
deployments and capabilities had continued to change since the program 
began and remained unclear. Further, the SPO did not have an approved 
integrated master schedule to guide the execution of the program, and our 
assimilation of available information indicated that key milestones 



GAO Has Previously 
Reported on Numerous 
SBlnet Acquisition 
Management Weaknesses 
and Risks 



See for example, GAO, Secure Border Initiative: DHS Needs to Reconsider Its Proposed 
Investment in Key Technology Program, GAO-10-340 (Washington, D.C.: May 5, 2010); 
Secure Border Initiative: DHS Needs to Address Testing and Performance Limitations 
That Place Key Technology Program at Risk, GAO-10-158 (Washington, D.C.: Jan. 29, 
2010); and DOD Business Systems Modernization: Important Management Controls 
Being Implemented on Major Navy Program, but Improvements Needed in Key Areas, 
GAO-08-896 (Washington, D.C.: Sept. 8, 2008). 

9 GAO, Secure Border Initiative: SBlnet Expenditure Plan Needs to Better Support 
Oversight and Accountability , GAO-07-309 (Washington, D.C.: Feb. 15, 2007). 

10 GAO, Secure Border Initiative: DHS Needs to Address Significant Risks in Delivering 
Key Technology Investment, GAO-08-1086 (Washington, D.C.: Sept. 22, 2008). 
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continued to slip. This schedule-related risk was exacerbated by the 
continuous change in and the absence of a clear definition of the approach 
used to define, develop, acquire, test, and deploy SBlnet. Accordingly, we 
concluded that the absence of clarity and stability in these key aspects of 
SBlnet impaired the ability of Congress to oversee the program and hold 
DHS accountable for results, and it hampered DHS's ability to measure the 
program's progress. 

• SBlnet requirements had not been effectively defined and managed. While 
the SPO had issued guidance that defined key practices associated with 
effectively developing and managing requirements, the guidance was 
developed after several key activities had been completed. In the absence 
of this guidance, the SPO had not effectively performed key requirements 
development and management practices, such as ensuring that different 
levels of requirements were properly aligned. As a result, we concluded 
that the risk of SBlnet not meeting mission needs and performing as 
intended was increased, as were the chances of the system needing 
expensive and time-consuming rework. 

• SBlnet testing was not being effectively managed. For example, the 
program office had not tested the individual system components to be 
deployed to initial locations, even though the contractor had initiated 
integration testing of these components. Further, its test management 
strategy did not contain, among other things, a clear definition of testing 
roles and responsibilities or sufficient detail to effectively guide planning 
for specific test events. 

To address these issues, we recommended that DHS assess and disclose 
the risks associated with its planned SBlnet development, testing, and 
deployment activities and that it address the system deployment, 
requirements management, and testing weaknesses that we had identified. 
DHS largely agreed to implement our recommendations. 

In September 2009, we reported that SBlnet had continued to experience 
delays. 11 For example, deployment to the entire southwest border had 
slipped from early 2009 to 2016, and final acceptance of TUS-1 and AJO-1 
had slipped from November 2009 and March 2010 to December 2009 and 



GAO, Secure Border Initiative: Technology Deployment Delays Persist and the Impact 
of Border Fencing Has Not Been Addressed, GAO-09-896 (Washington, D.C.: Sept. 9, 2009). 
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June 2010, respectively. We did not make additional recommendations at 
that time. 

In January 2010, we reported that SBlnet testing was not being effectively 
managed. 12 Specifically, we reported the following: 

• Test plans, cases, and procedures for the most recent test events were not 
defined in accordance with important elements of relevant guidance. 
Further, a large percentage of the test cases 13 for these events were 
changed extemporaneously during execution. While some of the changes 
were minor, others were more significant, such as rewriting entire 
procedures and changing the mapping of requirements to test cases. 
Moreover, these changes to procedures were not made in accordance with 
documented quality assurance processes, but rather were based on an 
undocumented understanding that program officials said they established 
with the contractor. Compounding the number and significance of changes 
were questions raised by the SPO and a support contractor about the 
appropriateness of some changes, noting that some changes made to 
system qualification test cases and procedures appeared to be designed to 
pass the test instead of being designed to qualify the system. 

• About 1,300 SBlnet defects had been found from March 2008 through July 
2009, with the number of new defects identified during this time generally 
increasing faster than the number being fixed — a trend that is not 
indicative of a system that is maturing and ready for deployment. While 
the full magnitude of these unresolved defects was unclear because the 
majority were not assigned a priority for resolution, some of the defects 
that had been found were significant. Although DHS reported that these 
defects had been resolved, they had nevertheless caused program delays, 
and related problems had surfaced that continued to impact the program's 
schedule. 

In light of these weaknesses, we recommended that DHS (1) revise the 
program's overall test plan; (2) ensure that test schedules, plans, cases, 
and procedures are adequately reviewed and approved consistent with the 
revised test plan; (3) ensure that sufficient time is provided for reviewing 
and approving test documents prior to beginning a given test event; and 



12 GAO-10-158. 

13 Atest case is a set of test inputs, execution conditions, and expected results developed 
for a particular objective, such as to verify compliance with a specific requirement. 
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(4) triage the full inventory of unresolved system problems, including 
identified user concerns, and periodically report on their status to CBP 
and DHS leadership. DHS fully agreed with the last three 
recommendations and partially agreed with the first. 

In May 2010, we raised further concerns about the program and called for 
DHS to reconsider its planned investment in SBlnet. 14 Specifically, we 
reported the following: 

• DHS had defined the scope of the first incremental block of SBlnet 
capabilities that it intended to deploy and make operational; however, 
these capabilities and the number of geographic locations to which they 
were to be deployed continued to shrink. For example, the geographic 
"footprint" of the initially deployed capability has been reduced from three 
border sectors spanning about 655 miles to two sectors spanning about 
387 miles. 

• DHS had not developed a reliable integrated master schedule for 
delivering the first block of SBbief. Specifically, the schedule did not 
sufficiently comply with seven of nine key practices that relevant guidance 
states are important to having a reliable schedule. For example, the 
schedule did not adequately capture all necessary activities, assign 
resources to them, and reflect schedule risks. As a result, it was unclear 
when the first block was to be completed, and continued delays were 
likely. 

• DHS had not demonstrated the cost-effectiveness of Block 1. In particular, 
it had not reliably estimated the costs of this block over its entire life 
cycle. Further, DHS had yet to identify expected quantifiable and 
qualitative benefits from this block and analyze them relative to costs. 
Without a meaningful understanding of SBlnet costs and benefits, DHS 
lacks an adequate basis for knowing whether the initial system solution is 
cost effective. 

• DHS had not acquired the initial SBlnet block in accordance with key life 
cycle management processes. While processes associated with, among 
other things, requirements development and management and risk 
management, were adequately defined they were not adequately 
implemented. For example, key risks had not been captured in the risk 



14 GAO-10-340. 
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management repository and thus had not been proactively mitigated. As a 
result, DHS is at increased risk of delivering a system that does not 
perform as intended. 

We concluded that it remains unclear whether the department's pursuit of 
SBlnet is a cost-effective course of action, and if it is, that it will produce 
expected results on time and within budget. Accordingly, we 
recommended that DHS (1) limit near-term investment in the first 
incremental block of SBlnet, (2) economically justify any longer-term 
investment in SBlnet, and (3) improve key program management 
disciplines. DHS largely agreed with our recommendations, and noted 
ongoing and planned actions to address each of them. 

In June 2010, we reported several technical, cost, and schedule challenges 
facing SBlnet, such as problems with radar functionality, and noted that 
the program office was staffed substantially below planned levels for 
government positions. 15 We did not make any additional recommendations 
at that time. 



Federal regulations and relevant guidance recognize effective contractor 
management and oversight as a key acquisition management discipline. In 
addition, they describe a number of practices associated with it, including 
training persons who are responsible for contractor management and 
oversight, verifying and deciding whether to accept contract deliverables, 
conducting technical reviews with the contractor to ensure that products 
and services will satisfy user needs, and holding management reviews with 
the contractor to monitor contractor performance. 16 

To DHS's credit, it has largely defined key practices aimed at employing 
each of these contractor management and oversight controls. Moreover, it 
has implemented some of them, such as training key contractor 
management and oversight officials and holding management reviews with 
the prime contractor. However, it has not effectively implemented others, 
such as documenting contract deliverable reviews and using entry and exit 
criteria when conducting technical reviews. Reasons for these weaknesses 



15 GAO, Department of Homeland Security: Assessments of Selected Complex 
Acquisitions, GAO-10-588SP (Washington, D.C.: June 30, 2010). 

16 See, for example, Federal Acquisition Regulation, part 46, quality assurance; and SEI, 
CMMI" for Acquisition. 



DHS Has Largely 
Defined but Has Not 
Effectively 
Implemented the Full 
Range of Needed 
Contractor 
Management and 
Oversight Controls 
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include limitations in the defined verification and acceptance deliverable 
process and a SPO decision to exclude some deliverables from the 
process, and insufficient time to review technical review documentation. 
Without employing the full range of practices needed to effectively manage 
and oversee the prime contractor, DHS is limited in its ability to know 
whether the contractor is meeting performance and product expectations. 
Moreover, it increases the chances that SBlnet will not function as 
intended and will take more time and resources than necessary to deliver, 
which we have previously reported is the case for Block 1. 



Training supports the successful performance of relevant activities and 
tasks by helping to ensure that the responsible people have the necessary 
skills and expertise to perform the tasks. According to relevant guidance, 
organizations should define training expectations and should ensure that 
these expectations are met for individuals responsible for contractor 
management and oversight. 17 

DHS's acquisition management directives define training requirements for, 
among others, program managers, contracting officers, and contracting 
officer's technical representatives (COTR). 18 Specifically: 

• Program managers must be certified at a level commensurate with their 
acquisition responsibilities. For a Level 1 information technology 
program, 19 like SBlnet, the designated program manager must be certified 
at a program management Level 3. 20 



SBlnet Contractor 
Management and 
Oversight Officials Have 
Been Properly Trained 



"SEI, CMMf for Acquisition. 

18 Training requirements for key officials are defined in three DHS policies: (1) Acquisition 
Certification Requirements for Program Manager, Management Directive (MD) 0782 (May 
26, 2004); (2) Contracting Officer Warrant Program (Apr. 16, 2007); and (3) COTR 
Certification, Appointment & Responsibilities, MD 0780.1 (Dec. 20, 2004). 

19 DHS has categorized major information technology investments as Levels 1, 2, and 3. A 
Level 1 major investment is defined in DHS Acquisition Directive 102-01 as program at or 
more than $1 billion in life cycle costs. 

20 DHS's Acquisition Certification Requirements for Program Manager, MD 0782 (May 26, 
2004) establishes three levels of certification: Level 1 establishes the foundation for 
managing increasingly complex investments or portfolios, Level 2 includes knowledge of 
strategic and capital planning, and Level 3 represents mastery of the knowledge and skills 
associated with the complexities of managing DHS's most strategic and sophisticated 
acquisitions. 
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• Contracting officers must be certified at the appropriate level to support 
their respective warrant authority. 21 

• COTRs must be trained and certified within 60 days of appointment to a 
contract or task order. The minimum mandatory requirements are the 
completion of 40 hours of COTR training and a 1-hour procurement ethics 
training course. 

For SBlnet, CBP has ensured that people in each of these key positions 
have been trained in accordance with DHS requirements. Specifically, the 
two program managers associated with the development and deployment 
of SBlnet Block 1 — the SBI Executive Director and the SPO Executive 
Director — were issued training certifications that qualify each as an 
Acquisition Professional. Further, each contracting officer responsible for 
executing actions on the Arizona Deployment Task Order (ADTO), the 
Design Task Order (DTO), the C3I/COP task order, and the System Task 
Order (STO) between June 2008 and February 2010 was certified 
commensurate with his or her respective warrant authority. In addition, 
for the same time period, each COTR assigned to each of the four task 
orders received DHS-issued certificates indicating that each had met the 
minimum training requirements before being assigned to a task order. 

According to CBP officials, DHS leadership has made contractor 
management and oversight training a high priority, which helped to ensure 
that key officials were trained. By doing so, CBP has established one of the 
key controls to help ensure that prime contractor products and services 
meet expectations. 



DHS Has Not Effectively 
Verified and Accepted All 
Contract Deliverables 



Effectively implementing a well-defined process for verifying and 
accepting prime contractor deliverables is vital to SBlnef s success. DHS 
has a defined process for verifying and accepting contract deliverables, 
but this process does not ensure that deliverable reviews are sufficiently 
documented. Further, while the SPO has followed key aspects of this 
process, it has not effectively documented its review of certain 
deliverables and has not effectively communicated to the prime contractor 



For warrant authority up to $100,000, the contracting officer must have a Federal 
Acquisition Certification in Contracting (FAC-C) Program Level I certification; for warrant 
authority up to $25 million, the contracting officer must have a FAC-C Level II certification; 
and for warrant authority more than $25 million, including unlimited warrant authority, the 
contracting officer must have a FAC-C Level III certification. 
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the basis for rejecting all deliverables. Reasons for not doing so include 
limitations in the defined verification and acceptance process and a SPO 
decision to exclude some deliverables from the process. Without 
documenting all its reviews and effectively communicating the review 
results to the contractor, the SPO has increased the chances of accepting 
deliverables that do not meet requirements and having the contractor 
repeat work to correct deliverable problems. 

CBP Has a Defined Process for The purpose of contractor deliverable verification and acceptance is to 
Verifying and Accepting SBlnet ensure that contractor-provided products and services meet specified 
Contract Deliverables requirements and otherwise satisfy the terms of the contract. According to 

relevant guidance, organizations should have written policies and 
procedures for verifying and accepting deliverables that, among other 
things, (1) assign roles and responsibilities for performing verification and 
acceptance tasks and (2) provide for conducting and documenting 
deliverable reviews and for effectively communicating to the contractor 
deliverable acceptance and rejection decisions. 22 

To its credit, CBP has defined policies and procedures for verifying and 
accepting SBlnet deliverables. Specifically, it issued its Deliverable Review 
and Approval Process in July 2007, which specifies how the SPO is to 
receive, review, and respond to all contract deliverables. Among other 
things, this guide assigns verification and acceptance roles and 
responsibilities to key program management positions. For example, it 
assigns the project manager 23 responsibility for overseeing the review 
process and determining the acceptability of the deliverables, designates 
reviewers 24 responsibilities for examining the deliverable, and assigns the 
contracting officer responsibility for communicating the decision to 
accept or reject the deliverable to the contractor. In addition, it provides 
for conducting deliverable reviews, which according to program officials, 
involves comparing the deliverable to the requirements enumerated in the 
applicable task order statement of work (typically within the Data Item 



22 SEI, CMMI® for Acquisition. 

23 A project manager is responsible for each of the task orders. These project managers have 
overall responsibility and accountability for achieving the task order's scope, cost, 
schedule, and technical objectives. 

24 According to the SPO, subject matter experts review the deliverables based on the 
contractual requirements that are outlined in the Data Item Descriptions, as well as then- 
own expertise and knowledge about the deliverables. 
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Description). The process further specifies that the decision to accept or 
reject the deliverable is to be communicated in writing to the contractor. 



However, the process does not state that the results of all reviews are to 
be documented. Instead, its states that a deliverable review comment form 
is to be prepared only when deficiencies or problems exist. If the 
deliverable is acceptable, the form does not need to be prepared. Program 
officials could not explain why review documentation was not required for 
acceptable deliverables. As a result, and as discussed in the next section of 
this report, the SPO cannot demonstrate its basis for accepting a number 
of deliverables, which in turn has increased the risk of, and actually 
resulted in, deliverables being accepted that do not meet requirements. 



CBP Has Not Fully Followed its 
Defined Process for Verifying 
and Accepting SBlnet 
Deliverables 



The SPO followed its process for verifying and accepting SBIree£-related 
deliverables about 62 percent of the time, based on the 29 deliverables that 
we reviewed. 25 Specifically, the process was fully followed for 18 of the 
deliverables: (1) 6 that were accepted without documented review 
comments, (2) 5 that were accepted with documented review comments, 
and (3) 7 that were rejected with documented review comments. In 
addition, the acceptance or rejection of all 18 deliverables was 
communicated in writing to the contractor. For example, the ADTO 
Security Plan Addendum was delivered to the SPO in August 2008. The 
SPO reviewed the plan and documented its review comments, which 
included a determination that the plan did not address all required items 
specified in the task order's Data Item Description. The CBP contracting 
officer subsequently notified the contractor in writing that the plan was 
rejected for this and other reasons. In February 2009, the contractor 
resubmitted the plan, the SPO reviewed the plan and documented its 
comments, and the contracting officer again notified the contractor in 
writing that the plan was rejected. The contractor resubmitted the plan in 
May 2009, and program officials documented their review of the 
deliverable. The contracting officer subsequently communicated the 
deliverable's acceptance in writing to the contractor. (Figure 3 summarizes 
the number of contract deliverables that did and did not follow the defined 
process.) 



25 These 29 deliverables actually resulted in 46 separate submissions because, for example, 
an initial submission was rejected and had to be resubmitted. 
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Figure 3: Summary of Contract Deliverables That Did and Did Not Follow the Defined Process 



Deliverables following process 



Accepted with comments and with letter 

Accepted without comments and with 
letter 

Rejected with comments and with letter 



Contract deliverables 



Deliverables not following process 




Accepted without comments and without 
letter 

Rejected without comments and without 
letter 

Decision not made, no comments, and no 
letter 



Source: GAO analysis of DHS data. 



The remaining 11 deliverables, however, did not fully adhere to the defined 
process. Of these, five were accepted without any documented review 
comments and without communicating the acceptance in writing to the 
contractor. The following are examples of these five and reasons for not 
adhering to the process. 



• Three of the five deliverables related to the C3I/COP task order did not 
require government approval. However, the Deliverable Review and 
Approval Process document does not provide for such treatment of these 
deliverables, and thus this practice is not in accordance with the process. 
Program officials told us that they have since recognized that this was a 
poor practice, and they have modified the task order to now require 
approval of all C3I/COP task order deliverables. 

• One of the five deliverables relates to the STO and is for the C2 
Component Qualification Test package, which includes, among other 
things, the test plan and test cases and procedures for conducting the C2 
qualification test event. In this case, the SPO accepted the test plan 
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because, according to program officials, several days had passed since the 
deliverable was received, and they had not received any comments from 
the reviewers. They said that they therefore accepted the deliverable on 
the basis of not receiving any review comments to the contrary, but did 
not notify the contractor in writing of the acceptance. 

The 11 deliverables also include 3 that were rejected without documented 
review comments and without the rejection being communicated to the 
contractor in writing. The following are examples of these three and 
reasons for not adhering to the process are discussed below. 

• One of the three deliverables relates to the C3I/COP task order and is for 
the Network Operations Center/Security Operations Center (NOC/SOC) 26 
Test Plan/Procedures/Description. According to program officials, the 
contractor did not submit the plan on time, thus requiring them to review 
it during the readiness review. 27 Based on this review, the plan was 
rejected, which was communicated verbally to the contractor during the 
review. Despite rejecting the plan, the program office began testing the 
NOC/SOC component on the day of the review, without a revised plan 
being submitted, reviewed, and approved. According to program officials, 
this occurred in part because of insufficient time and resources to review 
contractor test-related deliverables. 

• Another one of the three deliverables also relates to the C3I/COP task 
order, and is for the Release 0.5 Software Test Plan/Procedures/ 
Description. According to program officials, the contractor submitted the 
plan late. The program office rejected the plan and provided oral 
comments during a teleconference prior to the review. Nevertheless, the 
test event again occurred without a revised plan being submitted, 
reviewed, and accepted. According to program officials, this was also due 
to insufficient time and resources to review the test plan. In this case, the 
plan was approved in late April 2009, which was 5 months after the test 
event was conducted. 



26 The NOC/SOC monitors networked equipment, provides alerts for network problems, 
protects equipment from network-based attacks, and provides user authentication. 

27 According to the C3I/COP task order statement of work, the plan was to be submitted 10 
days before the test readiness review. However, the contractor submitted the plan the day 
of this review. 
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The 11 deliverables also include 3 for which a decision to accept or reject 
the deliverable was not made. See the following examples: 

• One of the three relates to the C3I/COP task order, and is for the NOC/SOC 
Interface Control Document. For this deliverable, review comments were 
not documented and no written communication with the contractor 
occurred. The deliverable was subsequently submitted three times and 
ultimately accepted. However, program officials could not explain 
whether the initial deliverable was accepted or rejected, or why the 
deliverable was submitted multiple times. 

• Another one of these relates to STO, and is for the Dynamic Object- 
Oriented Requirements System. 28 For this deliverable, review comments 
were not documented, but CBP communicated in writing to the contractor 
that it was withholding comment on this submission of the deliverable and 
was to provide a consolidated set of comments on the subsequent 
submission. Subsequently, the contractor resubmitted the deliverable, and 
because it was accepted, the review was not documented. The contracting 
officer communicated the deliverable's acceptance in writing to the 
contractor. 

By not effectively verifying and accepting contractor deliverables, the SPO 
cannot ensure that the deliverables will satisfy stated requirements, thus 
increasing the risk of costly and time-consuming rework. For example, we 
recently reported that contractor-delivered test plans were poorly defined 
and resulted in problems during testing. In particular, NOC/SOC testing 
was hampered by requirements incorrectly mapped to test cases, did not 
provide for testing all requirements, and required significant 
extemporaneous changes to test cases during the test events. 29 As a result 
of the testing problems, the SPO had to conduct multiple test events. 



Technical reviews are performed throughout the project life cycle to 
confirm that products and services being produced by the contractor 
provide the desired capability and ultimately satisfy user needs. To its 
credit, DHS has defined a process for conducting technical reviews, but it 
has not effectively implemented it. In particular, the SPO did not ensure 
that all key documentation was reviewed and relevant criteria were 



28 This system is the program office's requirements management database. 
29 GAO-10-158. 



Key Contract Technical 
Reviews Have Not Been 
Adequately Conducted 
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satisfied before concluding key technical reviews. Program officials 
attributed these limitations to the program's aggressive schedule, which 
resulted in insufficient time to review relevant documentation. Concluding 
technical reviews without adequate justification has resulted in schedule 
delays and costly rework. 

CBP Has a Defined Process for According to relevant guidance, organizations should have written policies 
Conducting Technical Reviews procedures for conducting technical reviews that, among other things, 

(1) assign roles and responsibilities for performing the specific technical 
review tasks and (2) establish entry and exit criteria to determine the 
readiness of the technical solution to proceed to the technical review and 
to demonstrate and confirm completion of required accomplishments. 30 

To its credit, DHS has policies and guidance for conducting technical 
reviews. Specifically, DHS's Systems Engineering Life Cycle outlines the 
key reviews to be performed as well as how these reviews are aligned with 
the department's governance process. 31 In addition, DHS guidance defines 
expectations for technical review exit criteria, stating that compliance 
with exit criteria is based upon the satisfaction of the content of the 
criteria, and not upon only the delivery of specified documents. 

To augment DHS policy and guidance, the SBlnet Systems Engineering 
Plan (SEP), dated November 2008, identifies and describes the technical 
reviews to be conducted. They include, for example: 

• Requirements Review, which is to ensure that requirements have been 
completely and properly identified and are understood by the SPO and the 
contractor. Documentation associated with this review is to include, 
among other things, a requirements traceability matrix (i.e., a tool for 
demonstrating that component-level requirements are traceable to higher- 
level system level requirements). 

• Critical Design Review (CDR), which is to (1) demonstrate that the 
designs are complete and baselined and (2) ensure that the solution is 
ready for fabrication, coding, assembly, and integration. Documentation 
for this review is to include, among other things, (1) baselined 
requirements, (2) interface descriptions, and (3) identified risks and 
mitigation plans. 

30 SEI, CMMI® for Acquisition. 

31 DHS, Acquisition Instruction/Guidebook, 102-01-001, Appendix B, Systems 
Engineering Life Cycle, Interim Version 1.9 (Nov. 7, 2008). 
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• Test Readiness Review, which is to assess the readiness of the system 
solution to begin formal testing. Documentation for this review is to 
include, among other things, (1) test plans that include test cases and 
procedures and (2) a traceability matrix that maps each requirement to be 
tested to a corresponding test case. 

In addition, the SBlnet SEP describes high-level roles and responsibilities 
for performing these reviews, and establishes entry and exit criteria for 
each. For example, it states that the SPO program manager and the chief 
engineer are responsible for leading the reviews. Further, the SEP defines 
entry and exit criteria for the CDR. For example: 

• Entry. System-level requirements should be traceable to component-level 
requirements; system internal and external interfaces should be defined. 

• Exit. Design baseline should be established and balanced across cost, 
schedule, performance, and risk considerations over the investment's life 
cycle; system risks should be identified and mitigation plans should be in 
place. 



Contractor Technical Reviews The SPO did not follow the defined process for conducting technical 
Have Not Been Performed reviews. Instead, program officials told us that they used the requirements 

Adequately defined in the respective task orders to guide each review. However, the 

task orders do not define entry and exit criteria. Rather, they list a set of 
documents that the contractor is to provide and the SPO is to review. For 
example, for the Block 1 CDR, the relevant task order requires that the 
contractor deliver, among other documents, (1) baselined component and 
system requirements, (2) interface descriptions (i.e., descriptions of the 
data to be exchanged and the protocols used to exchange the data), and 
(3) all identified risks and mitigation plans for those risks. However, the 
task orders do not associate these documents with either entry or exit 
criteria, and they do not specify characteristics or qualities that the 
documents are to satisfy. 

Without explicit entry and exit criteria, the basis for beginning and ending 
the technical reviews is unclear, thus increasing the risk that a program 
will be allowed to proceed and begin the next phase of development 
before it is ready to do so. In fact, this risk was realized for SBlnet. 
Technical reviews were concluded without adequate justification, which 
ultimately resulted in problems that required additional time and 
resources to fix. For example: 
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• NOC/SOC Requirements Review. At this review, the contractor did not 
deliver a requirements traceability matrix, as required by the relevant task 
order, until almost a month after the review was completed. Nonetheless, 
program officials stated that they concluded the review in June 2008, 
without knowing whether the applicable higher-level system requirements 
were fully satisfied. 

• Block 1 CDR. For this review, the contractor delivered (1) the baselined 
component and system requirements, (2) the interface descriptions, and 
(3) all identified risks and mitigation plans for those risks. 

However, these deliverables did not demonstrate that all component-level 
requirements were baselined and interface descriptions were understood. 
As we previously reported, baselined requirements associated with the 
NOC/SOC were not adequately defined at the time of the CDR, as 
evidenced by the fact that they were significantly changed 2 months later. 3 
Program officials stated that while they knew that requirements were not 
adequately baselined at this review, they believed that the interface 
requirements were understood well enough to begin system development. 
However, this was also not the case. Specifically, 39 of 90 NOC/SOC 
interface requirements were removed from the baseline, and 2 new 
interface requirements were added after CDR. 

Further, all relevant risks were not identified, and not all identified risks 
had mitigation plans. Specifically, 7 of 31 identified risks did not have 
mitigation plans, including risks associated with poorly established 
requirements traceability and inadequately defined requirements for 
integration suppliers. Moreover, the risks identified were as of May 2008, 
prior to the beginning of CDR, and did not include four risks identified 
between June and October 2008, when CDR was concluded. For example, 
a risk associated with the instability of the C3I/COP software was not 
addressed during CDR. 

Without properly baselined requirements (including interfaces) and 
proactive mitigation of known risks, system performance shortfalls are 
likely. To illustrate, we previously reported that ambiguities in 
requirements actually forced testers to rewrite test steps during execution 
based on interpretations of what they thought the requirements meant, 



M GAO-10-340. 
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and they required the SPO to incur the time and expense of conducting 
multiple events to test NOC/SOC requirements. 33 

• NOC/SOC Component Qualification Test Readiness Review. The 

SPO did not ensure that a well-defined test plan was in place, to include, 
among other things, test cases and procedures and a traceability matrix 
that maps each requirement to be tested to a corresponding test case. 
Specifically, the contractor delivered the test plan on the day of the 
review, rather than 10 days prior to the review, as required by the relevant 
task order. Nevertheless, the SPO concluded the review based on its 
review of the plan during the test readiness review. In this regard, we 
previously reported problems with the NOC/SOC test plan, noting that the 
plan mapped 28 out of 100 requirements to incorrect test cases. Program 
officials attributed the test plan limitations to, among other things, 
insufficient time and resources to review the deliverables. 

The SBlnet independent verification and validation (IV&V) contractor 34 
also identified weaknesses within technical reviews. Specifically, the IV&V 
contractor reported that the SPO was not provided with documentation, 
including the test plan, early enough for the NOC/SOC test readiness 
review to allow sufficient time for review. Moreover, in December 2009, 
the program identified technical oversight of technical review milestones 
as a major risk to the cost, schedule, and performance goals of the 
program. According to program officials, they are developing a technical 
review manual that is to supplement the SEP and provide detailed 
guidance for conducting technical reviews. In commenting on a draft of 
this report, DHS stated that it plans to complete and implement its 
technical review guide by December 2010. 

Program officials attributed the limitations with the technical reviews, in 
large part, to the program's aggressive schedule, which resulted in 
insufficient time to review relevant documentation. As discussed above, 
these limitations have resulted in schedule delays and costly rework. 



33 GAO-10-158. 

34 In 2008, the SPO contracted with an IV&V agent to, among other things, review the 
program's test documentation, execution, and related processes. Generally, the purpose of 
IV&V is to independently determine the extent to which program processes and products 
meet quality standards. The use of FV&V is recognized as an effective practice for large and 
complex system development and acquisition programs, like SBIwet 
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Contractor Management 
Reviews Were Conducted 
as Planned, but Were 
Limited by Unreliable 
Contractor Performance 
Data 



Management reviews help to ensure that the contractor's interpretation 
and implementation of the requirements are consistent with those of the 
program office. According to relevant guidance, organizations should have 
written policies and procedures for conducting management reviews that, 
among other things, (1) involve relevant stakeholders; (2) assign roles and 
responsibilities for performing management review tasks; (3) 
communicate project status information, including cost and schedule 
information, and risks; and (4) identify, document, and track action items 
to closure. 36 



CBP policy also recognizes the importance of these reviews by requiring 
the conduct of management reviews. 36 Further, the SBlnet Program 
Management Plan identifies the types of management reviews that are to 
be conducted with the contractor. For SBlnet, the primary management 
review is known as the Joint Program Management Review (JPMR). The 
plan also identifies, for example, the stakeholders that are to participate in 
the reviews, including the program manager, project managers, program 
control staff, and the risk management team; and it specifies the topics 
that are to be discussed at the reviews, such as project status, cost and 
schedule performance, and risks. 

To its credit, the program office has held monthly JPMRs, and these 
reviews include the prime contractor, the SBI Executive Director, the 
SBlnet Program Manager, program control staff, risk management team, 
border patrol agents, DHS representatives, and the Defense Contract 
Management Agency (DCMA). Further, review documentation shows that 
JPMRs addressed cost and schedule performance to include EVM 
performance data, project milestones status and potential delays, and 
risks. Review documentation also shows action items were identified, 
documented, and tracked to closure, as part of these reviews. For 
example, during the May 2009 JPMR, an action to review the risk 
management process — including roles, responsibilities, and decision 
authority — was identified, and this action was documented in Boeing's 
Management Emphasis Tracking database. 37 To address and close the 
action item, the program subsequently completed a review of the SBlnet 



*SEI, CMMT for Acquisition. 

36 CBP, System Life Cycle Handbook, Version 1.2 (Sept. 30, 2008). 

37 This database is Boeing-managed and tracks SBlnet action items throughout their 
lifecycle. The database captures such information as individual action assignments, status 
and updates, and relevant comments. 
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risk management process and tool, including reviewing lessons learned 
from other programs. The results of the review were presented during a 
February 2010 briefing, and the action item was closed. 

Effectively conducting management reviews has helped to provide 
program leadership with an understanding of the contractor's progress 
and the program's exposure to risks so that appropriate corrective action 
can be taken and the chances of delivering a system solution that meets 
mission needs within budget are enhanced. However, as discussed in the 
next section, the EVM performance data presented at these management 
reviews were not reliable, thus rendering those reviews, at best, limited in 
the extent to which they disclosed the true status of the program. 



DHS Has Not 
Effectively Monitored 
the Contractor's 
Progress in Meeting 
Cost and Schedule 
Expectations 



Measuring and reporting progress against cost and schedule expectations 
(i.e., baselines) is a vital element of effective contractor management and 
oversight. As noted earlier, EVM provides a proven means for measuring 
progress against cost and schedule commitments and thereby identifying 
potential cost overruns and schedule delays early, when the impact can be 
minimized. 

However, DHS has not ensured that its prime contractor's EVM system, 
which was certified as meeting relevant standards, has been effectively 
implemented on SBlnet. In particular, it has not ensured that performance 
measurement baselines were validated in a timely manner, that established 
baselines were complete and realistic, and that contractor-provided cost 
and schedule data were reliable. Reasons cited by program officials for 
these weaknesses include the instability in the scope of the work to be 
performed, an unexpected temporary stop in Block 1 design and 
deployment work when SBlnet funding was redirected, and the 
contractor's use of estimated, rather than actual, costs for subcontractor 
work, which are subsequently adjusted when actual costs are received. 
Without effectively implementing EVM, DHS has not been positioned to 
identify potential cost and schedule problems early, and thus has not been 
able to take timely actions to correct problems and avoid program 
schedule delays and cost increases. 
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Contractor EVM System In August 2005, the Office of Management and Budget issued guidance 1 
Has Been Certified that, among other things, directs agencies to ensure that EVM systems are 

compliant with the American National Standards Institute (ANSI) 
standard. 2 The ANSI standard consists of 32 guidelines associated with a 
sound EVM system that are intended to ensure that data are reliable and 
can be used for informed decision-making. 

The program office relies on the prime contractor's EVM system to 
provide cost and schedule performance data. This system was certified in 
April 2005 by DCMA as being compliant with the ANSI standard. DCMA 
certified the contractor's EVM system again in February 2009. 

Notwithstanding these certifications, DCMA identified a number of issues 
with the contractor's implementation of its EVM system. 3 In particular, in 
January 2010, DCMA reported that the SBlnet prime contractor's 
implementation of EVM was not consistent with all of the 32 ANSI 
guidelines. Specifically, DCMA identified concerns with the quality of 
scheduling and reporting, and the identification of significant differences 
between planned and actual cost and schedule performance, as well as 
reasons for those differences. 



Program Office Has Not 
Established Timely 
Performance Baselines 



According to relevant guidance, the performance measurement baseline, 
which is the foundation of an EVM system and the estimated cumulative 
value of planned work, serves as the value against which performance is 
measured for the life of the program or task order. 4 As such, it should be 
established as early as possible after contract or task order award, or 
whenever a major contract modification or baseline change occurs. DHS 



Office of Management and Budget Memorandum, M-05-23 (Aug. 4, 2005). 
2 ANSI/EIA-748-B. 

3 As mentioned previously, DCMA conducts periodic surveillance reviews of the prime 
contractor's EVM system implementation. 

4 GAO-09-3SP, 227-229, and Department of Defense, Earned Value Management 
Implementation Guide (October 2006). 
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guidance further states that a baseline should be validated within 90 days 
of the contract or task order award. 42 

However, the program office validated a performance measurement 
baseline within 90 days for only two of the six baselines that we reviewed 
(see figure 4). For the other four, the length of time to establish a validated 
baseline ranged from 5 to 10 months. For example, the program office 
issued the ADTO in June 2008, and it did not establish a validated baseline 
until 10 months later in April 2009. Similarly, in February 2009, the program 
office modified the scope of the STO and extended the period of 
performance, but it did not validate the revised baseline to include the 
additional scope and time until 7 months later in September 2009. Figure 4 
summarizes the periods of time during which earned value was, and was 
not, measured against a validated baseline. 



Figure 4: Summary of Time Frames With and Without Validated Performance Baselines from June 2008 through February 2010 
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Source: GAO analysis of SBInef data. 

Notes: Generally, the government established baselines through the period of performance for the 
task order at the time of the Integrated Baseline Review (IBR). The government had not completed all 
work for the DTO and STO within the original period of performance, and as such, extended the 
period of performance, and in some cases, established a new baseline. 

According to program officials, they held an IBR for the C3I/COP task order in March 2009. However, 
they did not provide sufficient documentation to support this statement. 



DHS, Department of Homeland Security Acquisition Manual, Section 3034.202 
Integrated Baseline Reviews (October 2009). Integrated baseline reviews are intended to 
verify that the baseline is realistic and the scope, schedule, and risks are mutually 
understood by the contractor and the government. 
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According to program officials, the delays in validating performance 
baselines were due to instability in the work to be performed, and the 
need to temporarily stop Block 1 design and deployment work between 
September 2008 and January 2009 because of DHS's decision to redirect 
funds from SBlnet to the physical infrastructure. 43 Without validated 
baselines, DHS was not positioned to identify potential cost and schedule 
problems early and to take timely corrective actions to mitigate those 
problems. 



DHS Has Not Established 
Reliable Performance 
Measurement Baselines 



An integrated baseline review (IBR) is used to validate the performance 
measurement baseline. This review is intended to verify that the baseline 
is realistic and ensure that the contractor and the government mutually 
understand scope, schedule, and risks for a given task order before a 
substantial amount of work is performed. According to relevant guidance, 
establishing a complete and realistic performance measurement baseline 
includes (1) assigning responsibility for managing, tracking, and reporting 
earned value data for work performed; (2) estimating needed resources 
(i.e., budgets and staff), including management reserve, for performing 
assigned tasks; (3) defining a product-oriented description of all work to 
be performed; 44 (4) scheduling all work in a time-phased sequence that 
reflects the duration of the program's activities; and (5) establishing 
objective performance measures for each task. 45 

In validating the performance measurement baselines for the four task 
orders that we reviewed, the program office implemented two of the above 
elements, but it did not implement the other three. Specifically, for each of 
the six baselines associated with the task orders, the program office (1) 
assigned responsibility for managing, tracking, and reporting earned value 
data associated with each work breakdown structure element and (2) 
estimated a time-phased budget, including the anticipated staff needed, for 
each work breakdown structure element, and established a management 
reserve. However, as discussed in the following section, the program 
office did not (1) define a product-oriented description of all work to be 



Physical infrastructure includes physical fencing and roads. 

44 The scope of the work is generally defined in a work breakdown structure, which is a 
hierarchical structure that shows how work tasks relate to one another as well as to the 
overall end product. 

45 GAO-09-3SP, 215-231. 
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performed, (2) reliably estimate schedule baselines, and (3) adequately 
measure earned value performance. 

Program officials attribute these limitations in establishing comprehensive 
baselines to instability in the nature of the work to be performed and the 
prime contractor's method for determining subcontractor performance. 
Nevertheless, without complete and realistic baselines, the SPO has been 
hampered in its ability to conduct meaningful measurement and oversight 
of the prime contractor's status and progress, as well as holding the 
contractor accountable for results. More importantly, the lack of 
meaningful measurement and oversight has contributed to program cost 
overruns and schedule delays. 

Work Breakdown Structure According to relevant guidance, a work breakdown structure deconstructs 

Was Not Product Oriented and a program's end product into successively smaller levels until the work is 
Did Not Include All Work subdivided to a level suitable for management control. 46 Further, a work 

breakdown structure should be product oriented and include all work to 
be performed. 

The work breakdown structure that was used to define each of the task 
order baselines was not product oriented. Instead, it was defined in terms 
of functions that span multiple system products, such as systems 
engineering, system test and evaluation, and program management. 

Additionally, the work breakdown structure did not reflect all work to be 
performed. Specifically, for four of the six performance measurement 
baselines, the work breakdown structure did not include all work 
described in the corresponding task order's statement of work. For 
example, the work breakdown structure used to define the May 2008 STO 
baseline did not include the work associated with identifying and selecting 
components that meet system requirements and program security. 
Similarly, DCMA reported in June 2008 that the work breakdown structure 
included in this baseline did not account for all work identified in the 
system task order. 

A reliable schedule provides a road map for systematic execution of a 
program and the means by which to gauge progress, identify and address 
potential problems, and promote accountability. Our research has 
identified nine best practices associated with developing and maintaining 



Estimated Baseline Schedules 
Were Not Reliable 



4,J GAO-09-3SP, 215. 
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a reliable schedule: (1) capturing all activities, (2) sequencing all activities, 
(3) assigning resources to all activities, (4) establishing the duration of all 
activities, (5) integrating activities horizontally and vertically, (6) 
establishing the critical path 47 for all activities, (7) identifying reasonable 
"float" 48 between activities, (8) conducting a schedule risk analysis, and (9) 
updating the schedule using logic and durations. 49 

The six task order baselines were not reliable because they substantially 
complied with only two of the eight key schedule estimating practices, and 
they did not comply with, or only partially or minimally complied with, the 
remaining six practices. 50 (See figure 5 for a summary of the extent to 
which each of the baseline schedules met each of the eight practices.) 



The critical path represents the chain of dependent activities with the longest total 
duration in the schedule. 

48 Float is the amount of time a task can slip before affecting the critical path. 
49 GAO-09-3SP, 218-224. 

°°We did not assess the ninth practice because the schedules that were reviewed at the 
IBRs were the initial schedules for each task order, and as such, were not updated. 
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Figure 5: Summary of IBR Schedules' Satisfaction of Schedule Estimating Practices 
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Met = DHS provided complete evidence that satisfied the criteria 

Substantially Met = DHS provided evidence that satisfied more than one-half of the criteria 
^) Partially Met = DHS provided evidence that satisfied about one-half of the criteria 

Minimally Met = DHS provided evidence that satisfied less than one-half of the criteria 
(^) Not Met = DHS provided no evidence that satisfied any of the criteria 
Source: GAO analysis of DHS data. 



Capturing all activities. The six schedules did not capture all activities 
defined in the task order baseline. Specifically, five of the six schedules 
did not reflect the work to be performed across the four task orders (i.e., 
integrated master schedule). Further, as previously mentioned, four of six 
work breakdown structures were missing elements defined in the 
respective task order statements of work. Moreover, two of the six 
schedules did not reflect all work that was defined in the work breakdown 
structure. For example, the December 2009 DTO schedule omitted efforts 
associated with design work for TUS-1 and AJO-1. 
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• Sequencing all activities. The six schedules substantially met this 
practice. Each of the schedules identified almost all of the predecessor 
and successor activities. However, each contained improper predecessor 
and successor relationships. For example, the May 2008 STO baseline 
included 52 of 538 activities (about 10 percent) with improper predecessor 
and successor relationships. Additionally, many activities in four of the 
schedules were constrained by "start no earlier than" dates. For example, 
as previously reported, the September 2009 baseline schedule contained 
403 of 1,512 activities (about 27 percent) with "start no earlier than" 
constraints, which means that these activities are not allowed to start 
earlier than their assigned dates, even if their respective predecessor 
activities have been completed. 

• Assigning resources to all activities. Two of the six schedules partially 
met this practice. Specifically, two schedules included resources; 
however, those resources were allocated to less than 15 percent of the 
activities identified in each schedule. Moreover, the remaining four 
schedules did not include estimated resources. Instead, resources for all 
six schedules were maintained separately as part of the contractor's 
earned value system and only available to DHS upon request. 

• Establishing the duration of all activities. Each of the six baseline 
schedules substantially met this practice. Specifically, each schedule 
established the duration of key activities and included baseline start and 
end dates for most of the activities. Further, reasonable durations were 
established for the majority of the activities in the schedules, meaning that 
the durations established were less than 44 days. Nevertheless, each of the 
schedules included activities that were not of short duration, that is, more 
than 44 days. For example, the April 2009 ADTO baseline included 29 of 
1,009 activities with durations ranging from 45 days to 352 days. 

• Integrating activities horizontally and vertically. Each of the 
schedules partially met this practice. As mentioned previously, the six 
schedules did not capture all activities defined in the task order baseline. 
Further, four of six work breakdown structures were missing elements 
defined in respective task order statements of work. Additionally, five of 
six schedules did not reflect the work performed across the four task 
orders (i.e., integrated master schedule), and each had improper 
predecessor and successor relationships. 

• Establishing the critical path for all activities. Each of the six 
schedules partially met this practice. Specifically, four of six work 
breakdown structures were missing elements defined in the respective 
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task order statements of work. Additionally, four of the six schedules were 
missing predecessor and successor activities, and each of the schedules 
included improper predecessor and successor relationships. Further, five 
of the six schedules did not reflect the work to be performed across the 
four task orders (i.e., integrated master schedule). Unless all activities are 
included and properly linked, it is not possible to generate a true critical 
path. 

• Identifying reasonable float between activities. Each of the 
schedules identified float; however, the amount of float was excessive. For 
example, the February 2008 C3I/COP task order baseline included 259 of 
294 activities (about 88 percent) with float greater than 100 days and 189 
of the 259 (about 73 percent) with float in excess of 200 days. 

• Conducting a schedule risk analysis. DHS did not conduct a risk 
analysis of any of the schedules. 

Earned Value Performance Was According to the ANSI standard for EVM systems, 51 only work for which 
Not Adequately Measured measurement is impractical may be classified as "level-of-effort." 52 Our 

research shows that if more than 15 percent of a program's budget is 
measured using level-of-effort, then that amount should be scrutinized 
because it does not allow schedule performance to be measured (i.e., 
performance equals planned work). 53 

However, the six baselines had between 34 and 85 percent of the baseline 
dollar value categorized as level-of-effort, including four with more than 
50 percent (see table 2). Moreover, for five of the six baselines, program 
documentation showed that the program office did not identify any action 
items during the respective IBRs related to the high use of level-of-effort. 

According to program officials, the STO, which categorized between 70 
and 85 percent of the baseline dollar value as level-of-effort, includes many 
program management activities (e.g., cost, schedule, and subcontractor 
management). Nevertheless, they recognized that the level-of-effort for 
this task order was high, and in November 2009, they directed the 



-"ANSI/EIA-748-B. 

52 Level-of-effort is unmeasured effort of a general or supportive nature which does not 
produce definitive end products (e.g., program administration). 

53 GAO-09-3SP, 226. 
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contractor to minimize the use of level-of -effort for STO. According to 
program officials, the high level-of-effort was due, in part, to the prime 
contractor's use of this measurement for subcontractor work. In 
November 2009, DCMA stated that the SPO's use of level-of-effort 
activities was high, noting that this could be masking true contractor 
performance. 



Table 2: Percentage of Performance Measurement Baseline Work Tasks 
Categorized as Level-of-Effort for Six Baselines 



Task order IBR 


Level-of-effort percentage 


C3I/COP task order February 2008 
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STO May 2008 


85 percent 


DTO March 2009 


53 percent 


ADTO April 2009 


56 percent 


STO September 2009 


70 percent 


DTO December 2009 


34 percent 



Source: GAO analysis of DHS data. 



Unreliable EVM Data Limit 
DHS's Ability to Measure 
Contractor Performance 



If performed properly, EVM can provide an objective means for measuring 
program status and forecasting potential program cost overruns and 
schedule slippages so that timely action can be taken to minimize their 
impact. To do so, however, the underlying EVM data must be reliable, 
meaning that they are complete and accurate and all data anomalies are 
explained. 



EVM Data Can Provide 
Objective Insight into Program 
Cost and Schedule 
Performance 



In the case of SBlnet, the EVM data provided by the prime contractor for 
the 21-month period ending in February 2010 have not been reliable, as 
evidenced by numerous and unexplained anomalies in monthly EVM 
reports. Reasons for the anomalies include the contractor's use of 
estimated, rather than actual, costs for subcontractor work, which are 
subsequently adjusted when actual costs are received. Without reliable 
performance data, the true status of the SBlnet program is unclear, thus 
limiting the SPO's ability to identify potential cost and schedule shortfalls. 

EVM is a proven program measurement approach that, if implemented 
appropriately, can create a meaningful and coherent understanding of a 
program's true health and status. As a result, the use of EVM can alert 
decision makers to potential program problems sooner than is possible by 
using actual versus planned expenditure alone, and thereby reduce the 
chance and magnitude of program cost overruns and schedule slippages. 
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Simply stated, EVM measures the value of completed work in a given 
period (i.e., earned value) against (1) the actual cost of work completed 
for that period (i.e., actual cost) and (2) the value of the work that is 
expected to be completed for that period (i.e., planned value). Differences 
in these values are referred to as cost and schedule variances, 
respectively. 

• Cost variances compare the value of the work completed with the actual 
cost of the work performed. For example, if a contractor completed $5 
million worth of work and the work actually cost $6.7 million, there would 
be a negative $1.7 million cost variance. 

• Schedule variances are also measured in dollars, but they compare the 
value of the work completed with the value of the work that was expected 
to be completed. For example, if a contractor completed $5 million worth 
of work at the end of the month but was expected to complete $10 million 
worth of work, there would be a negative $5 million schedule variance. 

Positive variances indicate that activities are costing less or are being 
completed ahead of schedule. Negative variances indicate activities are 
costing more or are falling behind schedule. To determine both cost and 
schedule variances, all three values are necessary. 

SBlnet EVM Data Have Not According to relevant guidance, EVM data should be valid and free from 

Been Reliable unexplained anomalies (e.g., missing or negative values) because they can 

limit program management's ability to identify potential cost and schedule 
shortfalls. 54 Therefore, anomalies should be minimized for each of the 
three values — earned value, planned value, and actual cost. Moreover, all 
anomalies should be identified, and the reason for each should be fully 
explained in the monthly EVM reports. To do less limits the completeness 
and accuracy of these values, and thus makes the resulting variance 
determinations unreliable. While an industry standard for what constitutes 
an acceptable volume of anomalies does not exist, EVM experts in the 
public and private sectors that we interviewed stated that the occurrence 
of EVM data anomalies should be rare. Of these experts, some agreed that 
an anomaly should occur in no more than 5 percent of the work 
breakdown structure elements for a given contract or task order, while 
some of these advocated an occurrence percentage of no more than 1-2 
percent. 



o4 GAO-09-3SP, 226. 
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However, the EVM data that the prime contractor delivered to the SPO 
from June 2008 through February 2010 (21 months) contained numerous, 
unexplained anomalies. Specifically, the monthly EVM reports for all four 
task orders that we reviewed showed one or more anomalies (e.g., missing 
or negative values for earned value, planned value, and actual cost) in 
each of the months that had a validated performance measurement 
baseline. More specifically, the average number of work breakdown 
structure elements across the four task orders that had data anomalies 
during this 21-month period ranged from 11 percent to 41 percent. For the 
C3I/COP task order in particular, the monthly percentage of work 
breakdown structure elements with anomalies ranged between 25 and 67 
percent over the 21 months. (See figure 6 for the percentage of work 
breakdown structure elements with anomalies by month for each of the 
four task orders.) 



Figure 6: Percentage of Work Breakdown Structure Elements with EVM Data Anomalies by Month for Each of the Four Task 
Orders 
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Source: GAO analysis of DHS data. 

Note: We only analyzed EVM reports for those months with a validated baseline for any of the four 
task orders. There were no validated baselines for any of the four task orders in March 2009. 
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The October 2009 STO monthly EVM report illustrates how the anomalies 
can distort the contractor's performance. According to this report, about 
$13,000 worth of work was planned to be completed on integration and 
management, approximately $13,000 worth of work was performed, and 
actual costs were about negative $550,000. Thus, the report erroneously 
suggests that the contractor performed $13,000 worth of work, and 
actually saved about $550,000 in doing so. Similarly, the September 2009 
ADTO monthly report showed that about $200,000 worth of work was 
planned to be completed on tower sites and infrastructure, and $25,000 
worth of work was performed, but that no costs were incurred, suggesting 
that the work was performed for free. 

Exacerbating the large percentage of monthly data anomalies across the 
four task orders is the fact that in most cases the reasons for the 
anomalies were not explained in the monthly EVM variance analysis 
reports. Specifically, about 79 percent of all anomalies across all four task 
orders during the 21-month period were not explained. In particular, 82 of 
119 (or about 69 percent) of all data anomalies for the STO task order 
were not explained in the monthly reports, and none of the anomalies 
were explained for DTO. (See figure 7 for the total number of data 
anomalies and the number that were explained in the monthly reports 
across the four task orders.) 
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Figure 7: Total Number of Data Anomalies and Number of Explained Anomalies in the Monthly EVM Reports across the Four 
Task Orders 

Number 
45 



30 




June July Aug. Sept. Oct. Nov. Dec. Jan. Feb. Mar. Apr. May June July Aug. Sept. Oct. Nov. Dec. Jan. Feb. 



2008 2009 2010 



| | Total EVM data anomalies 

| Total EVM data anomalies with explanations 
Source: GAO analysis of DHS data. 

Note: As mentioned previously, there were no validated baselines for any of the four task orders in 
March 2009. 

Program officials acknowledged problems with the EVM data and stated 
that they meet with the prime contractor each month to discuss the EVM 
reports, including the reliability of the data. According to program 
officials, limitations in the EVM data are due, in part, to the contractor's 
use of estimated, rather than actual, costs for subcontractor work, which 
are subsequently adjusted when actual costs are received. Officials further 
stated that they have been working with the contractor to reduce the 
volume of unexplained anomalies, and they believe that the reliability of 
the data has improved since February 2010. However, program officials 
did not provide any documentation to support this statement. Without 
reliable EVM data, the program office is unable to identify actual cost and 
schedule shortfalls, which along with the other contractor tracking and 
oversight weaknesses discussed in this report, has limited its ability to 
effectively minimize program cost increases and schedule delays. 
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Effective management and oversight of a program's prime contractor is 
essential to successfully acquiring and deploying a system like SBlnet. 
Integral to accomplishing this is defining and implementing a range of 
contractor management and oversight controls (e.g., processes and 
practices) that reflect relevant federal guidance and best practices. To do 
less increases the chances that contractor-delivered products and services 
will not satisfy stated requirements and will not meet customer 
expectations. The result is incurring the additional time and expense to 
redo or rework contractor deliverables, accepting products and services 
that do not perform as intended and do not meet mission needs, or both. 

Overall, DHS has not done an effective job of managing and overseeing its 
prime contractor, including monitoring the contractor's performance. DHS 
has largely defined key management and oversight processes and 
practices that it should have followed, and it implemented a number of 
these processes and practices. However, several key management and 
oversight controls were not adequately defined, and essential controls 
were not implemented. Most significantly, DHS did not adequately 
document deliverable reviews and communicate the basis for rejecting 
certain deliverables in writing to the contractor, which contributed to 
deliverables that did not live up to expectations and necessitated rework 
and caused later problems. Further, technical reviews were not grounded 
in explicit criteria for determining when reviews should begin and 
conclude, which also contributed to contract deliverables requiring costly 
and time-consuming rework. In addition, the cost and schedule baselines 
for measuring the contractor's performance were frequently validated too 
late and without sufficient accuracy and completeness to provide a 
meaningful basis for understanding performance, which precluded DHS 
from taking timely action to correct unfavorable results and trends. 
Compounding these serious baseline limitations was contractor-provided 
data about actual performance that were replete with unexplained 
anomalies, thus rendering the data unfit for effective contractor 
management and oversight. Notwithstanding of a number of contractor 
management and oversight definition and implementation efforts that DHS 
executed well, such as defining key processes and practices and training 
key staff, these above-cited weaknesses collectively mean that DHS's 
management and oversight of its prime contractor has been a major 
contributor to the SBlnet program's well-chronicled history of not 
delivering promised system capabilities on time and on budget. 

These limitations can be attributed to a number of factors, including gaps in 
how certain processes and practices were defined, as well as not enforcing 
other processes and practices that were defined and applicable and not 
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taking sufficient time to review deliverables that were submitted late. The 
limitations can be further attributed to the fact that SBlnet has from its 
outset lacked clear definition and stability, and thus experienced continuous 
change in scope and direction — an issue that we have previously reported 
and made recommendations to address. Collectively, these factors have 
helped to create a contractor management and oversight environment, 
which, when combined with the many other acquisition management 
weaknesses that we have previously reported about and made 
recommendations to address, have produced a program that to date has not 
been successful, and if not corrected, can become worse. 



R6COmm6nd.cltiori.S for ^° ™P rove ^HS management and oversight of the SBlnet prime 

contractor, we recommend that the Secretary of Homeland Security direct 

Executive Action the Commissioner of the U.S. Customs and Border Protection to have the 

SBI Executive Director, in collaboration with the SBlnet Program 
Director, take the following four actions: 

• Revise and implement, as applicable, contractor deliverable review 
processes and practices to ensure that (1) contractor deliverables are 
thoroughly reviewed and are not constrained by late contractor 
deliverables and imposed milestones, (2) the reviews are sufficiently 
documented, and (3) the acceptance or the rejection of each contractor 
deliverable is communicated in writing to the contractor, to include 
explicit explanations of the basis for any rejections. 

• Ensure that applicable entry and exit criteria for each technical review are 
used and satisfied before initiating and concluding, respectively, a given 
review. 

• Establish and validate timely, complete, and accurate performance 
measurement baselines for each new task order or major modification of 
an existing task order, as appropriate, to include, but not be limited to, 
ensuring that (1) the work breakdown structure includes all work to be 
performed, (2) baseline schedules reflect the key schedule estimating 
practices discussed in this report, and (3) level-of-effort performance 
measurement in excess of 15 percent is scrutinized, justified, and 
minimized. 

• Ensure that all anomalies in contractor-delivered monthly earned value 
management reports are identified and explained, and report periodically 
to DHS acquisition leadership on relevant trends in the number of 
unexplained anomalies. 
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Because we have already made recommendations in prior reports to 
address the other management and oversight weaknesses discussed in this 
report, such as those related to requirements management, risk 
management, and Systems Engineering Plan implementation, we are not 
making any additional recommendations at this time. 



In written comments on a draft of this report, signed by the Director, 
Departmental GAO/OIG Liaison Office and reprinted in appendix II, DHS 
agreed with our four recommendations and described actions under way 
or planned, which we summarize below, to address them. 

• With respect to our recommendation to revise and implement the 
contractor deliverable review process, DHS stated that it is updating the 
process to require written documentation of each review and the 
communication to the contractor of review results. 

• With respect to our recommendation to ensure that entry and exit criteria 
are used to initiate and conclude each technical review, DHS stated that it 
has established an SBI Systems Engineering Directorate to focus on 
technical oversight of the acquisition process, adding that the Directorate 
is developing a technical review guide that describes in detail the review 
process and the relevant entry and exit criteria for each technical review. 

• With respect to our recommendation to establish and validate timely, 
complete, and accurate performance measurement baselines, DHS stated 
that it is mindful of the need to establish and maintain current 
performance baselines, and to plan and implement baseline updates as 
completely and promptly as practicable, which it indicated is done through 
IBRs. DHS also noted that while scheduling practices remain a challenge, 
it continues to make improvements to its process, including implementing 
scheduling tools and templates. 

• With respect to our recommendation to identify and explain all anomalies 
in monthly EVM reports, and to report periodically relevant trends to DHS 
acquisition leadership, DHS acknowledged the need to correctly document 
anomalies in the monthly EVM reports, and stated that it is working with 
DCMA to improve contractor quality control issues and the content of the 
monthly EVM reports. It also stated that it is augmenting the reports with 
routine conversations between contractor and project management staff. 
The department also committed to advising the appropriate acquisition 
leaders through established reporting and oversight opportunities as 
issues arise with contractor performance or reporting. 



Agency Comments 
and Our Evaluation 
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Notwithstanding its agreement with our recommendations, the department 
also commented that it took exception to selected findings and 
conclusions regarding the program's implementation of EVM. A summary 
of DHS's comments and our responses are provided below. 

• The department stated that it took exception to our finding that it did not 
ensure performance measurement baselines were validated in a timely 
manner, and said that it was not accurate to conclude that the lack of 
validated baselines precluded the program office from identifying cost and 
schedule problems and taking corrective action. In support of these 
positions, the department made the following three points, which our 
response addresses. 

First, the department stated that the SBlnet program office delayed formal 
IBRs until it had finalized negotiated modifications to the task orders, and 
in doing so, was able to complete an IBR within 90 days of each major task 
order modification. We do not question whether the program office held 
IBRs within 90 days of final negotiation of major task order modifications. 
Our point is that DHS did not validate task order performance 
measurement baselines (i.e., hold IBRs) within 90 days of task order 
award, which is what DHS guidance states should occur. As our report 
states, the program office only met this 90-day threshold for two of the six 
baselines that we reviewed. Further, the length of time to validate the 
performance baselines for the four task orders far exceeded 90 days (5 to 
10 months), during which time DHS reports show that significant work 
was performed and millions of dollars were expended. In fact, the DHS 
reports show that most of the planned work for some of these task orders 
had already been performed by the time the IBR was held and the baseline 
was validated. As we state in our report, and DHS acknowledged in its 
comments, the purpose of an IBR is to verify that the performance 
baseline is realistic and that the scope, schedule, and risks are mutually 
understood by the contractor and the government before a substantial 
amount of work is performed. 

Second, DHS commented that the program office maintained what it 
referred to as "interim" performance measurement baselines during the 
period of major program scope, schedule, and budget changes. We 
acknowledge that in some cases the program office had these "interim" 
baselines. However, these baselines are the contractor-provided baselines, 
meaning that the program office and the contractor had not mutually 
agreed to the scope, schedule, and risks associated with the work to be 
performed. Moreover, for two of the task orders, the program office did 
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not have an "interim" baseline, even though the contractor performed 
significant work under these task orders. 55 

Third, the department stated that program leadership reviewed the 
contractor's technical and financial performance information relative to 
performance measurement baselines and implemented management 
actions as needed. We do not question whether program leadership 
reviewed contractor-provided performance information or whether 
actions to address problems may have been taken. However, our report 
does conclude, as is discussed later in this section, that the combination of 
the EVM weaknesses that our report cites, to include unreliable 
performance baselines and contractor-provided performance data, did not 
allow the program office to identify performance problems early and to 
take timely actions to avoid the well-documented schedule delays and cost 
increases that the program has experienced. 

• The department expressed two concerns with how it said our report 
characterized and quantified EVM anomalies. 

First, the department stated that our report failed to distinguish between 
factual errors and legitimate monthly accounting adjustments. We agree 
that our report does not distinguish between the two types of anomalies, 
and would add that this was intentional because making the distinction 
was not relevant to our finding. Specifically, our finding is that the reasons 
for the numerous anomalies were not explained in the monthly EVM 
variance analysis reports, therefore making the true status of the program 
unclear. 

Second, DHS stated that we incorrectly concluded that both errors and 
adjustments are problematic, distort cost performance, and limit 
management insight. In response, we did not conclude that all errors and 
adjustments have these impacts, but rather that the lack of explanation 
associated with such a large volume of anomalies made the true status of 
the program unclear, thus limiting the program office's ability to identify 
actual cost and schedule shortfalls, which is certainly problematic. 
Further, our report cites examples of cost performance data that provide a 
distorted picture of actual performance vis-a-vis expectations. 
Accordingly, the correct characterization of the report's conclusion 



55 The program office did not have a baseline for ADTO and DTO, from July 2008 through 
October 2008 and from June 2008 through January 2009, respectively. 
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concerning the reliability of EVM data is that the lack of explanation of the 
numerous anomalies in monthly reports is problematic, provides a 
distorted picture of cost performance, and limits management insight. To 
this very point, DHS acknowledged in its comments the importance of 
explaining the reason for anomalies in the monthly variance reports, 
regardless of whether they are due to factual errors or accounting 
adjustments. 

• The department stated that it took exception to our conclusion that the 
program office's lack of validated baselines in particular, and EVM 
shortcomings in general, contributed to cost and schedule growth and 
made it unable to identify cost and schedule problems early and take 
corrective actions to avoid them. In response, we did not conclude that the 
lack of validated baselines alone had either of these impacts. However, we 
did conclude that the collection of EVM weaknesses discussed in our 
report, to include untimely validated baselines, incomplete and unreliable 
baselines, and unreliable performance data, together precluded the 
program office from identifying problems early and taking corrective 
action needed to avoid the program's well-chronicled history of schedule 
delays and cost increases. In support of this conclusion, we state in the 
report, for example, that the performance measurement baselines that we 
reviewed understated the cost and time necessary to complete the work 
because they did not capture all work in the task orders' statements of 
work and because they were not grounded in a range of scheduling best 
practices. Given that cost and schedule growth is a function of the 
baseline against which actual cost and schedule performance is measured, 
it follows logically that an understated baseline would produce actual cost 
overruns and schedule delays. In addition, we would note that beyond 
these EVM shortcomings, our report also recognizes other contract 
tracking and oversight, test management, and requirements management 
weaknesses that have collectively contributed to the program's cost, 
schedule, and performance shortfalls. 

In addition to the above points, DHS provided technical comments, which 
we have incorporated in the report as appropriate. 



We are sending copies of this report to the Chairmen and Ranking 
Members of the Senate and House Appropriations Committees and other 
Senate and House committees and subcommittees that have authorization 
and oversight responsibilities for homeland security. We will also send 
copies to the Secretary of Homeland Security, the Commissioner of U.S. 
Customs and Border Protection, and the Director of the Office of 
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Management and Budget. In addition, the report will be available at no 
cost on the GAO Web site at http://www.gao.gov. 

Should you or your offices have any questions on matters discussed in this 
report, please contact me at (202) 512-3439 or at hiter@gao.gov. Contact 
points for our Offices of Congressional Relations and Public Affairs may 
be found on the last page of this report. Key contributors to this report are 
listed in appendix III. 




Randolph C. Hite 

Director, Information Technology Architecture 
and Systems Issues 
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Appendix I: Objectives, Scope, and 
Methodology 



Our objectives were to determine the extent to which the Department of 
Homeland Security (DHS) (1) has defined and implemented effective 
controls for managing and overseeing the Secure Border Initiative 
Network (SBlnet) prime contractor and (2) is effectively monitoring the 
prime contractor's progress in meeting cost and schedule expectations. To 
accomplish our objectives, we focused on four key task orders — the 
Arizona Deployment Task Order, the Design Task Order, the System Task 
Order, and the Command, Control, Communications, and Intelligence/ 
Common Operating Picture Task Order — that are integral to the design, 
development, and deployment of the first increment of SBlnet, known as 
Block 1. We also focused on the SBlnet System Program Office's (SPO) 
contracting and oversight activities that occurred from June 2008 through 
February 2010. 

To determine the extent to which DHS has defined and implemented 
effective controls for managing and overseeing the SBlnet prime 
contractor, we focused on training, verifying and accepting contract 
deliverables, conducting technical reviews, and conducting management 
reviews. We chose these four because each is important to tracking and 
overseeing contractor performance for the four task orders. 

• Training. We compared relevant DHS contractor management and 
oversight training requirements to the program managers', contracting 
officers', and contracting officer's technical representatives' training 
certifications. 

• Verifying and accepting contract deliverables. We compared U.S. 
Customs and Border Protection's (CBP) process for verifying and 
accepting contract deliverables to leading industry practices 1 to identify 
any variances. We then assessed a nonprobability, random sample of 28 
contract deliverables that the SPO identified as being delivered between 
June 1, 2008, and August 31, 2009. 2 We also judgmentally selected one 
additional review that was conducted between September 1, 2009, and 



Software Engineering Institute, Capability Maturity Model Integration' for Acquisition, 
Version 1.2 (Pittsburgh, Pa., November 2007). 

2 We conducted a nonprobability sample because we determined that the SPO's contract 
deliverable log was not sufficiently reliable for purposes of identifying all required 
deliverables during our time frames, and thus we did not select a sample that could be 
projected to the total population. 
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Appendix I: Objectives, Scope, and 
Methodology 



February 28, 2010. 3 For each of the 29 deliverables, we reviewed relevant 
documentation, such as the contract deliverables, review comment forms, 
and documented communications with the prime contractor indicating 
acceptance or rejection of the deliverable, and compared them to the CBP 
process and leading industry practices to determine what, if any, 
deviations existed. 

• Technical reviews. We compared relevant DHS and CBP guidance and 
entry and exit criteria in the task order Data Item Descriptions to leading 
industry practices to identify any variances. We assessed a nonprobability, 
random sample of technical reviews. Specifically, we assessed a technical 
review from each of the eight unique combinations of task orders and 
review types held between June 1, 2008, and August 31, 2009. We also 
judgmentally selected one additional review that was conducted between 
September 1, 2009, and February 28, 20 10. 4 For each of the nine reviews, 
we compared the package of documentation prepared for and used during 
these reviews to the criteria defined in the relevant task orders to 
determine the extent to which the reviews satisfied the criteria. 

• Management reviews. We compared relevant CBP guidance to leading 
industry practices to identify any variances. We then compared relevant 
documentation prepared for and used during monthly joint program 
management reviews to determine the extent to which the reviews 
addressed cost, schedule, and program risks. We also assessed a 
nonprobability sample of 11 action items that were identified during the 
reviews held from October 2008 through October 2009, 5 and assessed 
relevant documentation to determine the extent to which they were 
tracked to closure. 

To determine the extent to which DHS is effectively monitoring the prime 
contractor's progress in meeting cost and schedule expectations, we 
focused on the program's implementation of earned value management 
(EVM) because it was the tool used to monitor the contractor's cost and 



3 We selected one additional deliverable that was associated with the technical review noted 
below to provide a more current assessment of the program's adherence to its process. 

4 We selected one additional technical review based on the combinations of task orders and 
review types in our existing sample to provide a more current assessment of the program's 
implementation of established criteria. 

5 We selected one action item for each month in which a joint program management review 
was held and action items were documented. 
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schedule performance. Specifically, we analyzed the six performance 
measurement baselines, and the associated integrated baseline review 
documentation, such as briefings, the work breakdown structure (WBS) 
governing all task orders, task order statements of work, schedules, 
monthly contract performance reports, control account plans, and 
responsibility assignment matrixes. In doing so, we compared this 
documentation to EVM and scheduling best practices as identified in our 
Cost Estimating and Assessment Guide* Specifically, for each of the six 
baselines: 

• We reviewed control account plans and responsibility assignment 
matrixes to determine the period of performance and scope of work for 
each baseline, compared the work described in the respective task order 
statements of work to the work described in the responsibility assignment 
matrix, and reviewed the control account plans to determine the extent to 
which the level-of-effort measurement method was to measure contractor 
performance. 

• We analyzed the schedule presented at each baseline review against eight 
key schedule estimating practices in our Cost Estimating and Assessment 
Guide. 7 In doing so, we used commercially available software tools to 
determine whether each schedule, for example, included all critical 
activities, a logical sequence of activities, and reasonable activity 
durations. Further, we characterized the extent to which the schedule met 
each of the practices as either not met, minimally met, partially met, 
substantially met, or met. 

• We analyzed the contract performance reports for each of the four task 
orders for each month that there was a validated baseline. Specifically, we 
identified instances of the following: (1) negative planned value, earned 
value, or actual cost; (2) planned value and earned value without actual 
cost; (3) earned value and actual cost without planned value; (4) actual 
cost without planned value or earned value; (5) earned value without 
planned value and actual cost; (6) inconsistencies between the estimated 



GAO, GAO Cost Estimating and Assessment Guide: Best Practices for Developing and 
Managing Capital Program Costs, GAO-09-3SP (Washington, D.C.: March 2009), 215 steps 
1-6. 

'We did not assess the ninth practice — updating the schedule using logic and durations — 
because the schedules that were reviewed at the integrated baseline reviews were the 
initial schedules for each task order, and as such, were not updated. 
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cost at completion and the planned cost at completion; (7) actual cost 
exceeding estimated cost at completion; and (8) planned or earned values 
exceeding planned cost at completion. To determine the number of 
anomalies, we identified each WBS element that had one or more of the 
above anomalies. Then, we identified the number of WBS elements at the 
beginning and the end of the baseline period of performance, and 
calculated the average number of WBS elements. We used this to 
determine the percentage of WBS elements with anomalies for each task 
order and for each month for which there was a validated baseline. 

• To support our work across this objective, we interviewed officials from 
the Department of Defense's Defense Contract Management Agency 
(DCMA), which provides contractor oversight services to the SPO, 
including oversight of EVM implementation, and prime contractor 
officials. We also reviewed DCMA monthly status reports and corrective 
action reports. 

For both objectives, we interviewed program officials to obtain 
clarification on the practices, and to determine the reasons for any 
deviations. 

To assess the reliability of the data that we used to support the findings in 
this report, we reviewed relevant program documentation to substantiate 
evidence obtained through interviews with knowledgeable agency 
officials, where available. We determined that the data used in this report 
are sufficiently reliable. We have also made appropriate attribution 
indicating the sources of the data. 

We performed our work at the CBP headquarters and prime contractor 
facilities in the Washington, D.C., metropolitan area and with DCMA 
officials from Huntsville, Alabama. We conducted this performance audit 
from June 2009 to October 2010 in accordance with generally accepted 
government auditing standards. Those standards require that we plan and 
perform the audit to obtain sufficient, appropriate evidence to provide a 
reasonable basis for our findings and conclusions based on our audit 
objectives. We believe that the evidence obtained provides a reasonable 
basis for our findings and conclusions based on our audit objectives. 
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of Homeland Security 



U.S. Department of Homdamd Security 
Washington, DC 20528 

Homeland 
1||F Security 



September 23, 2010 

Mr. Randolph C. Hite 

Director, Information Technology Architecture 
and Systems issues 

U.S. Government Accountability Office 
Washington, D. C. 20548 

Dear Mr. Hite: 

Thank you for the opportunity to review and offer comment on the Government Accountability Office 
(GAO) draft report entitled, "SECURE BORDER INITIATIVE: DHS Needs to Strengthen 
Management and Oversight of its Prime Contractor," GAO- 1 1-6, dated October 2010. GAO was 
asked to determine the extent to which the Department of Homeland Security (DHS) (1) defined and 
implemented effective controls for managing and overseeing die SBlnel prime contractor and (2) 
effectively monitored the prime contractor's progress in meeting cost and schedule expectations. To 
do this, GAO analyzed key performance documentation against relevant guidance and industry best 
practices, and interviewed program officials. 

GAO acknowledges that DHS has largely defined key management and oversight processes and 
procedures for verifying and accepting contract deliverables and conducting technical reviews. 
However, the draft report identifies measures that the U.S. Customs and Border Protection (CBP) 
SBInef can take to further strengthen contractor management and oversight. 

GAO made four recommendations to improve DHS management and oversight of the SBlner prime 
contractor. CBP concurs with the four recommendations. However, not withstanding our 
concurrence, we take exception to several assertions regarding aspects of the reliability of the Secure 
Border Initiative (SBI) program's Earned Value Management (EVM) practices, and government 
oversight of contractor activities. The comments below address our specific concerns. 

First, GAO asserts the program office did not ensure EVM performance measurement baselines 
(PMB) were "validated" in a timely manner, and concludes that the program office's EVM 
shortcomings led to cost and schedule growth in the program. We firmly take exception to these 
assertions. 

As background to many of the EVM findings and recommendations, GAO acknowledges throughout 
the report the significant amount of program change occurring with the SBlnet program in late 2008- 
2009. These positive changes followed a major program re-baseiining (Acquisition Program Baseline) 
with the Department as well as SBfete/ final system design updates after Critical Design Review. The 
program baseline change drove major scope, schedule, and budget changes across ail of the Boeing 
task orders and associated performance measurement baselines. 
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GAO concluded that without a "validated" baseline through the conduct of an Integrated Baseline 
Review (IBR), the program office %vas unable to identify cost and schedule problems early and to take 
timely corrective actions to mitigate those problems. The report also concluded that the program's 
lack of a "validated" baseline contributed to cost overruns and schedule delays. We believe these 
conclusions are inaccurate, and the causal assertion is not supported by any analysis presented in the 
report. Of significance is the fact thai the program office maintained interim performance 
measurement baselines throughout the transition period with the best available task plan information, 
as required by EVM best practices. Program leadership also reviewed in various forum, the 
contractor's technical and financial performance information relative to the established PMBs and 
implemented rigorous management actions as needed. The program office delayed formal IBR(s) 
until completing final, negotiated task order modifications, and in each case the IBR was completed 
within the timeframe specified in DHS directives (less than 90 days after the major contract 
modification). Certainly during a period of significant program changes, earned value management is 
challenged, but the program office believes prudent and compliant EVM measures were executed. 

GAO documented other EVM implementation concerns reported by the program office, specifically 
the prime contractor's over-reliance on the "level of effort" (LoE) EVM measure, and recurring 
quality control lapses, or "anomalies," contained in the contractor's monthly reports. For the LoE 
concern, the program office and the prime contractor initiated several actions to improve task order 
work planning by creating discreet work packages, enabling improved performance management and 
issue identification. However, given the stage of the program today, we will continue to see high 
levels of effort in the task order baselines due to maintaining sizeable contractor program management 
"overhead" (typically LoE), while transitioning from construction and testing completion into a 
sustaining system and software engineering mode (typically LoE), as well as performing a variety of 
system maintenance functions (also typically LoE). 

Secondly, we are concerned with GAO's characterization and quantification of "anomalies." The SB! 
program office demands accurate contractor performance reports, and is continuously engaged with 
the contractor to fix underlying problems. The draft report fails to distinguish between factual errors 
versus legitimate monthly accounting adjustments. For example — 

* Errors, and the resultant corrections, occur when the contractor collects and reports costs for 
one activity when the costs actually apply to another activity, and might include incorrect 
contract charge numbers issued to subcontractors or other simple administrative mistakes. 
Errors distort cost performance status however the SBI program office and the contractor 
promptly identify and take corrective actions to correct identified problems and improve 
quality controls for final monthly reports. 

• Adjustments routinely occur when the prime contractor reports estimated costs for a 
subcontractor (while awaiting a subcontractor to submit an invoice), and then in a subsequent 
report, the prime reconciles previously reported estimated costs with the final, invoiced actual 
costs. Adjustments may also be necessary to reflect contractor rate changes. These 
adjustments improve the accuracy and reliability of the cumulative cost performance data, yet 
may appear anomalous for the current month. The prime contractors adjustments are 
completed in accordance with the contractor's approved system guide and established best 
practices. 
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The GAO report combines both errors and adjustments when quantifying "anomalies" and incorrectly 
concludes that both errors and adjustments are problematic, distorting cost performance, and limits 
management insight. 

Regardless of error corrections or adjustments, the contractor is required to document such actions in 
the monthly cost performance report (Format 5). Over the past eight months, the contractor 
implemented quality control measures, and has significantly improved the process for monthly 
reporting of both errors and anomalies as well as significant accounting adjustments. 

The recommendations and CBP's corrective actions to address the recommendation are described 
below. 

Recomme ndation 1: Revise and implement, as applicable, contractor deliverable review processes 
and practices to ensure that ( 1 ) contactor deliverables are thoroughly reviewed and are not constrained 
by late contractor deliverables and imposed milestones, (2) the reviews are sufficiently documented, 
and (3) the acceptance or the rejection of each contractor deliverable is communicated in writing to the 
contractor, to include explicit explanations of the basis for any rejections. 

CBP Response: CBP concurs with the recommendation. Since the draft report was issued, we have 
taken several major steps to improve the SBlner program management structure, capabilities, 
procedures, and processes. This includes significant improvement in SBIne/ execution of the Contract 
Deliverable Requirements List (CDRL) review process. The program has commenced reviewing and 
updating the current CDRL review policy. The updated CDRL policy will document the process and 
procedures for program technical reviews, and written documentation to the contractor regarding 
results of the review to include reviewer comment forms and formal communication of the acceptance, 
conditional acceptance, or rejection of the contract deliverable. In implementing the updated CDRL 
policy, contract deliverable review participants will be trained to ensure that reviewers are technically 
qualified and possess the procedural knowledge to effectively review and provide concise and 
thorough, review comments and that the process is folly aligned with the approved CDRL review 
policy. 

Due Date : Policy update expected to be completed by December 2010. 

Recommendation 2: Ensure that applicable entry and exit criteria for each technical review are used 
and satisfied before initiating and concluding, respectively, a given review. 

CBP Response : CBP concurs with the recommendation. We have taken several major steps to 
improve the SBInef program management structure, capabilities, procedures, and processes. This has 
led to a more rigorous application of review entry and exit criteria. In addition, the SB! Systems 
Engineering (SE) Directorate was established to ensure that systems engineering capabilities are 
focused on providing the technical oversight required to support knowledge-based decision making 
throughout the acquisition process. The SE Directorate is incorporating the DHS Engineering Life 
Cycle best practices into a Technical Review Guide (TRG) that describes in detail the SE technical 
review process and the relevant entry and exit criteria for each technical review. The draft TRG is 
currently under review and when implemented will serve as the roadmap used by the SBi System 
Program Office to ensure that all relevant entry and exit criteria are understood and satisfied in a 
manner consistent with programmatic cost, schedule, and performance risk considerations. 
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Pue Date : Technical Review Guide expected to be completed and implemented by December 2010. 

Recommendation 3: Establish and validate timely, complete, and accurate performance measurement 
baselines for each new task order or major modification of an existing task order, as appropriate, to 
include, but not be limited to, ensuring that (I) the work breakdown structure includes all work to be 
performed; (2) baseline schedules reflect the key schedule estimating practices discussed in this report; 
and (3) level-of-effort performance measurement in excess of 15 percent is scrutinized, justified, and 
minimized. 

CBP Response: CBP concurs with the recommendation. The SB! program office is mindful of the 
need to establish and maintain current Performance Measurement Baselines (PMB), and to plan and 
implement baseline updates as completely and promptly as practicable. For new task orders and 
significant modifications, this is done through Integrated Baseline Reviews (IBRs). The purpose of 
the IBR is to confirm that the PMB covers the scope of work; the work is realistically and accurately 
scheduled; the proper mix of resources and budgets are assigned; the appropriate management 
processes are in place; and the risks associated with the PMB are identified, documented, and 
managed. Prior to an iBR, the program office routinely evaluates the control accounts, and the 
integrated PMB, for consistency, traceability, and completeness relative to the work breakdown 
structure (WBS), to include a cross-check of the WBS back to the control accounts. Also, the program 
office documents any adverse findings and resolves such with the prime contractor as part of the IBR 
proceedings. As described above, the program office scrutinizes, and with the prime contractor, 
minimizes level of effort tasks in the PMB. 

Scheduling practices remain a challenge, and as reported in our response to prior GAO reviews, we 
continue to upgrade our processes. The program office is revising our scheduling standards to reflect 
procedural improvements for schedule development, management and surveillance. We have created 
and put into practice the use of 26 diagnostic schedule filters to analyze the health of our schedules. 
The filters fully leverage GAO's scheduling best practices. We have also implemented improvements 
to our practices for developing integrated Master Plan/ Integrated Master Schedule that utilizes newly 
revised planning and schedule templates based on the DHS Systems Engineering Lifecycle. 

Due Date : Ongoing corrective actions are in place, and the program office continues to assess and 
manage contractor progress and improvements. 

Recommendation 4: Ensure that all anomalies in contractor-delivered monthly earned value 
management reports are identified and explained, and report periodically to DHS acquisition 
leadership on relevant trends in the number of unexplained anomalies. 

CBP Response : CBP concurs with the recommendation. The monthly contract performance report 
"anomalies" (which include a variety of legitimate monthly updates to actual costs, rate changes, etc.) 
need to be documented correctly each month. The SBi program office shared with the GAO 
significant, recent activities and progress with associated corrective actions: 

> Fixing prime contractor systemic and quality control issues, with the assistance of the 
Defense Contract Management Agency, through active participation in program EVM 
surveillance: and 

> improving discipline and content of Format 5 reporting narrative, augmented with routine 
management discussions with contractor and government project management staff. 
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Our focus remains to ensure we are able to determine program status relative to well-defined 
performance baselines. As issues arise with either contractor performance or reporting, we will advise 
the appropriate acquisition leaders through established reporting and oversight opportunities. 

Due Date : Ongoing; corrective actions are in place, and the program office continues to assess and 
manage contractor progress and improvements. 



Sincerely, 

Jeraid E. Levine 
Director 

Departmental GAO/OIG Liaison Office 
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